2021-11-20T08:38:41.777 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-20T08:38:41.779 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi --archive /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245 -- /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245/orig.config.yaml 2021-11-20T08:38:41.847 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-11-20T08:38:41.935 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245 branch: wip-yuri2-testing-2021-11-19-1714-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6515245' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2021-11-19-1714-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2021-11-19-1714-pacific sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2021-11-19-1714-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa targets: smithi093.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjRrWumVOQvvhOkdnPs44ou5Dbu31SnlsYqMvOx1Cj8w4DOtCf/BuMMkkMG5mAT/uIbHSBPk6PbmfB1Sb9UGfC+m5RmmrKnzm/4v6Z7+S18UibjpJ4PQNpLIptAqMLxtXweqFvAoiBgEUhPPDOlq2eouZZSGVcpvgSOEDPSFiNz82P5kWqvzIBPoQTmO1vciZNWTF1FgasqI2XGzMd5l+wNjN1vlv+3wpYNtsbDid+1CR8qW7CDBPsgngNIFS+OwM2JV6TXmBz28yHveownXT+FHLWmOg6XnUvba1zUVrXVeF/p186Zedc/PFCUMfGTHrt4OSHggm3OPsaBCfPJZS4BXtv8R3Q8yfo858TXzqPJ/ICvSMkAItw4hWMOLfjajWWL2w1MJkj2Jv5gWpJNc1/gBj68EGX8brcElsHiMS/S7pTkjgJRcitUj5+ufN6kn3GuM+IwKDgA2SQQ7XNBiBX7A1klpBzdYI3w/gb6AajJhLkjog1nte78M5kC3Dzv00= smithi102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0kpfTnUHZXi6zBUKp6rnfUzOBQQwZOKZ6vR5Y17zB4jABYols9OHW2GGADD+bAqVk65YpSw5rR24JRUfmRmDod4F5hu6Vqdug2u+H4HnYauS5h2fbNGTjGnosTADEoR1r7LPsd48+wodr5v6WMuUQAeY9Aepld0RPLOStomprSRKY7QaOTAYjRgbDYIvJ+yNpxiepYE/LdavDR1JAe6khgCU2OIHFqOqypTRo0DbfutZBHeX0ie7Go3yb/VEO9HAjMNbiolNh/KMXW8JdehS7c/gO8Vms5OqetrBFRr7bsUkaef6DsdNwbtnn2HKqtHnXYCCQGNuJQ3dtOYoxxDynGm4M0h3bfiY7QEroW6OD4N2yk8o+sPZyZ4nUB57z+sVvxpngmApssX21VsYjxxNi0H0WRrz5tEIvZ73ClYDmLNZqrxDo/h2hG+/Pu8DicNXKvJcy+raF0znW9CvSXv6IouSn+sgvyFPW/UCslzN9DbiO9PG0tUD13jj3S4mvDes= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: exclude_packages: - ceph-volume tag: v16.2.4 - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - ceph --format=json mds versions | jq -e ". | add == 4" - fs.pre_upgrade_save: null - kclient: null - print: '**** done kclient client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology_branch: master teuthology_sha1: 6cfddd5e1c7d5ede1367cf3087c21af44eb88356 timestamp: 2021-11-20_04:32:48 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2021-11-20T08:38:41.936 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa; will attempt to use it 2021-11-20T08:38:41.937 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks 2021-11-20T08:38:41.937 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-11-20T08:38:41.941 INFO:teuthology.task.internal:Checking packages... 2021-11-20T08:38:41.965 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa' 2021-11-20T08:38:41.966 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-11-20T08:38:41.967 INFO:teuthology.packaging:ref: None 2021-11-20T08:38:41.967 INFO:teuthology.packaging:tag: None 2021-11-20T08:38:41.967 INFO:teuthology.packaging:branch: wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T08:38:41.968 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T08:38:41.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T08:38:42.190 INFO:teuthology.task.internal:Found packages for ceph version 16.2.6-633.g9a97ad1b 2021-11-20T08:38:42.191 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-11-20T08:38:42.214 INFO:teuthology.task.internal:no buildpackages task found 2021-11-20T08:38:42.215 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-11-20T08:38:42.261 INFO:teuthology.task.internal:Saving configuration 2021-11-20T08:38:42.290 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-11-20T08:38:42.314 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-20T08:38:42.364 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi093.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 08:33:30.493056', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIN9S323cjXc86ev7siro5QMMTD9bJkC7rg3nEdleJRiqu3cRP/vIEBOtPFp+VAaPyaB6jgAhmkHBNQC22nwnyo='} 2021-11-20T08:38:42.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 08:33:30.491676', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvjerq/926tUBup05R6T6S4207Nnn3PYMYQLCT0HMlt4y7EpivuXrL1t3aG8d7NlCujqfT/wwPzNY1DlIr9YGM='} 2021-11-20T08:38:42.393 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-11-20T08:38:42.411 INFO:teuthology.task.internal:roles: ubuntu@smithi093.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-20T08:38:42.411 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-20T08:38:42.411 INFO:teuthology.run_tasks:Running task console_log... 2021-11-20T08:38:42.552 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6720f23598>, signals=[15]) 2021-11-20T08:38:42.554 INFO:teuthology.run_tasks:Running task internal.connect... 2021-11-20T08:38:42.576 INFO:teuthology.task.internal:Opening connections... 2021-11-20T08:38:42.577 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T08:38:42.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:38:42.707 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2021-11-20T08:38:42.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:38:42.824 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-11-20T08:38:42.839 DEBUG:teuthology.orchestra.run.smithi093:> uname -m 2021-11-20T08:38:42.860 INFO:teuthology.orchestra.run.smithi093.stdout:x86_64 2021-11-20T08:38:42.863 DEBUG:teuthology.orchestra.run.smithi093:> cat /etc/os-release 2021-11-20T08:38:42.920 INFO:teuthology.orchestra.run.smithi093.stdout:NAME="CentOS Stream" 2021-11-20T08:38:42.921 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION="8" 2021-11-20T08:38:42.921 INFO:teuthology.orchestra.run.smithi093.stdout:ID="centos" 2021-11-20T08:38:42.921 INFO:teuthology.orchestra.run.smithi093.stdout:ID_LIKE="rhel fedora" 2021-11-20T08:38:42.921 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_ID="8" 2021-11-20T08:38:42.922 INFO:teuthology.orchestra.run.smithi093.stdout:PLATFORM_ID="platform:el8" 2021-11-20T08:38:42.922 INFO:teuthology.orchestra.run.smithi093.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T08:38:42.922 INFO:teuthology.orchestra.run.smithi093.stdout:ANSI_COLOR="0;31" 2021-11-20T08:38:42.922 INFO:teuthology.orchestra.run.smithi093.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T08:38:42.923 INFO:teuthology.orchestra.run.smithi093.stdout:HOME_URL="https://centos.org/" 2021-11-20T08:38:42.923 INFO:teuthology.orchestra.run.smithi093.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T08:38:42.923 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T08:38:42.923 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T08:38:42.925 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server 2021-11-20T08:38:42.948 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2021-11-20T08:38:42.968 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2021-11-20T08:38:42.969 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2021-11-20T08:38:43.031 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2021-11-20T08:38:43.031 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="8" 2021-11-20T08:38:43.031 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2021-11-20T08:38:43.031 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2021-11-20T08:38:43.032 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="8" 2021-11-20T08:38:43.032 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el8" 2021-11-20T08:38:43.032 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T08:38:43.032 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2021-11-20T08:38:43.033 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T08:38:43.033 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2021-11-20T08:38:43.033 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T08:38:43.033 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T08:38:43.033 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T08:38:43.034 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2021-11-20T08:38:43.055 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-11-20T08:38:43.070 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-11-20T08:38:43.108 INFO:teuthology.task.internal:Checking for old test directory... 2021-11-20T08:38:43.109 DEBUG:teuthology.orchestra.run.smithi093:> test '!' -e /home/ubuntu/cephtest 2021-11-20T08:38:43.112 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2021-11-20T08:38:43.129 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-11-20T08:38:43.177 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-11-20T08:38:43.178 DEBUG:teuthology.orchestra.run.smithi093:> test -z $(ls -A /var/lib/ceph) 2021-11-20T08:38:43.182 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2021-11-20T08:38:43.224 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-11-20T08:38:43.291 INFO:teuthology.run_tasks:Running task kernel... 2021-11-20T08:38:43.308 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-11-20T08:38:43.309 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-11-20T08:38:43.310 DEBUG:teuthology.orchestra.run.smithi093:> uname -r 2021-11-20T08:38:43.311 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2021-11-20T08:38:43.327 INFO:teuthology.orchestra.run.smithi102.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T08:38:43.328 INFO:teuthology.task.kernel:Running kernel on smithi102: 4.18.0-301.1.el8.x86_64 2021-11-20T08:38:43.329 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2021-11-20T08:38:43.331 INFO:teuthology.orchestra.run.smithi093.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T08:38:43.331 INFO:teuthology.task.kernel:Running kernel on smithi093: 4.18.0-301.1.el8.x86_64 2021-11-20T08:38:43.332 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum install -y kernel 2021-11-20T08:38:45.879 INFO:teuthology.orchestra.run.smithi093.stdout:CentOS-8 - AppStream 59 MB/s | 17 MB 00:00 2021-11-20T08:38:46.038 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - AppStream 47 MB/s | 17 MB 00:00 2021-11-20T08:38:49.083 INFO:teuthology.orchestra.run.smithi093.stdout:CentOS-8 - Base 59 MB/s | 15 MB 00:00 2021-11-20T08:38:49.211 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Base 48 MB/s | 15 MB 00:00 2021-11-20T08:38:51.334 INFO:teuthology.orchestra.run.smithi093.stdout:CentOS-8 - Extras 235 kB/s | 16 kB 00:00 2021-11-20T08:38:51.535 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Extras 227 kB/s | 16 kB 00:00 2021-11-20T08:38:52.195 INFO:teuthology.orchestra.run.smithi093.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2021-11-20T08:38:52.487 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 13 MB/s | 11 MB 00:00 2021-11-20T08:38:54.571 INFO:teuthology.orchestra.run.smithi093.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-11-20T08:38:54.856 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 4.0 MB/s | 24 kB 00:00 2021-11-20T08:38:56.537 INFO:teuthology.orchestra.run.smithi093.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T08:38:56.537 INFO:teuthology.orchestra.run.smithi093.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T08:38:56.538 INFO:teuthology.orchestra.run.smithi093.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T08:38:56.841 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T08:38:56.841 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T08:38:56.841 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T08:38:56.860 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:38:56.861 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:38:56.862 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:38:56.862 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:38:56.862 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:38:56.862 INFO:teuthology.orchestra.run.smithi093.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T08:38:56.863 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T08:38:56.863 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T08:38:56.863 INFO:teuthology.orchestra.run.smithi093.stdout:Removing: 2021-11-20T08:38:56.863 INFO:teuthology.orchestra.run.smithi093.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T08:38:56.863 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T08:38:56.864 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T08:38:56.864 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:38:56.864 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:38:56.865 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:38:56.866 INFO:teuthology.orchestra.run.smithi093.stdout:Install 3 Packages 2021-11-20T08:38:56.866 INFO:teuthology.orchestra.run.smithi093.stdout:Remove 3 Packages 2021-11-20T08:38:56.866 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:38:56.866 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 74 M 2021-11-20T08:38:56.867 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:38:57.118 INFO:teuthology.orchestra.run.smithi093.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 27 MB/s | 7.0 MB 00:00 2021-11-20T08:38:57.158 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:38:57.159 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:38:57.160 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:38:57.160 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:38:57.160 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:38:57.161 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T08:38:57.161 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T08:38:57.161 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T08:38:57.161 INFO:teuthology.orchestra.run.smithi102.stdout:Removing: 2021-11-20T08:38:57.161 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T08:38:57.162 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T08:38:57.162 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T08:38:57.162 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:38:57.162 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:38:57.163 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:38:57.163 INFO:teuthology.orchestra.run.smithi102.stdout:Install 3 Packages 2021-11-20T08:38:57.163 INFO:teuthology.orchestra.run.smithi102.stdout:Remove 3 Packages 2021-11-20T08:38:57.163 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:38:57.164 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 74 M 2021-11-20T08:38:57.164 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:38:57.582 INFO:teuthology.orchestra.run.smithi102.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 17 MB/s | 7.0 MB 00:00 2021-11-20T08:38:57.600 INFO:teuthology.orchestra.run.smithi093.stdout:(2/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 40 MB/s | 30 MB 00:00 2021-11-20T08:38:58.376 INFO:teuthology.orchestra.run.smithi093.stdout:(3/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 25 MB/s | 38 MB 00:01 2021-11-20T08:38:58.379 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:38:58.381 INFO:teuthology.orchestra.run.smithi093.stdout:Total 49 MB/s | 74 MB 00:01 2021-11-20T08:38:58.519 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:38:58.580 INFO:teuthology.orchestra.run.smithi102.stdout:(2/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 26 MB/s | 38 MB 00:01 2021-11-20T08:38:58.788 INFO:teuthology.orchestra.run.smithi102.stdout:(3/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 18 MB/s | 30 MB 00:01 2021-11-20T08:38:58.788 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:38:58.789 INFO:teuthology.orchestra.run.smithi102.stdout:Total 46 MB/s | 74 MB 00:01 2021-11-20T08:38:58.919 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:38:59.159 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:38:59.160 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:38:59.592 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:38:59.593 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:39:00.293 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:39:00.315 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:39:00.793 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:39:00.988 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:39:02.983 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:39:03.458 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T08:39:03.854 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:39:04.347 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T08:39:07.719 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T08:39:08.107 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T08:39:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T08:39:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T08:39:11.166 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T08:39:11.336 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T08:39:11.337 INFO:teuthology.orchestra.run.smithi093.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T08:39:11.567 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T08:39:11.900 INFO:teuthology.orchestra.run.smithi093.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T08:39:11.940 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T08:39:12.134 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T08:39:12.135 INFO:teuthology.orchestra.run.smithi102.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T08:39:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T08:39:12.389 INFO:teuthology.orchestra.run.smithi102.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T08:39:13.706 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T08:39:13.908 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:13.916 INFO:teuthology.orchestra.run.smithi093.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:14.597 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T08:39:14.827 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:14.839 INFO:teuthology.orchestra.run.smithi102.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:16.775 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:17.697 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:37.611 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T08:39:37.982 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:37.983 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T08:39:37.983 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T08:39:37.983 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T08:39:37.983 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T08:39:37.984 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T08:39:38.310 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T08:39:38.481 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:38.482 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:39:38.482 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:39:38.482 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:38.483 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:38.483 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:38.483 INFO:teuthology.orchestra.run.smithi093.stdout:Removed: 2021-11-20T08:39:38.483 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T08:39:38.483 INFO:teuthology.orchestra.run.smithi093.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T08:39:38.484 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:39:38.484 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:39:38.601 DEBUG:teuthology.orchestra.run.smithi093:> rpm -q kernel --last | head -n 1 2021-11-20T08:39:38.687 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:38.687 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T08:39:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T08:39:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T08:39:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T08:39:38.688 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T08:39:38.802 INFO:teuthology.orchestra.run.smithi093.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 08:39:11 AM UTC 2021-11-20T08:39:38.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:38.804 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:38.804 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T08:39:38.804 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T08:39:38.805 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2021-11-20T08:39:38.805 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi093.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T08:39:38.805 DEBUG:teuthology.orchestra.run.smithi093:> rpm -q kernel --last | head -n 1 2021-11-20T08:39:38.947 INFO:teuthology.orchestra.run.smithi093.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 08:39:11 AM UTC 2021-11-20T08:39:38.948 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:38.949 DEBUG:teuthology.orchestra.run.smithi093:> sudo rpm -qi grub2-tools 2021-11-20T08:39:38.980 INFO:teuthology.orchestra.run.smithi093.stdout:Name : grub2-tools 2021-11-20T08:39:38.980 INFO:teuthology.orchestra.run.smithi093.stdout:Epoch : 1 2021-11-20T08:39:38.980 INFO:teuthology.orchestra.run.smithi093.stdout:Version : 2.02 2021-11-20T08:39:38.981 INFO:teuthology.orchestra.run.smithi093.stdout:Release : 99.el8 2021-11-20T08:39:38.981 INFO:teuthology.orchestra.run.smithi093.stdout:Architecture: x86_64 2021-11-20T08:39:38.981 INFO:teuthology.orchestra.run.smithi093.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T08:39:38.981 INFO:teuthology.orchestra.run.smithi093.stdout:Group : System Environment/Base 2021-11-20T08:39:38.982 INFO:teuthology.orchestra.run.smithi093.stdout:Size : 9264704 2021-11-20T08:39:38.982 INFO:teuthology.orchestra.run.smithi093.stdout:License : GPLv3+ 2021-11-20T08:39:38.982 INFO:teuthology.orchestra.run.smithi093.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T08:39:38.983 INFO:teuthology.orchestra.run.smithi093.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T08:39:38.983 INFO:teuthology.orchestra.run.smithi093.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T08:39:38.983 INFO:teuthology.orchestra.run.smithi093.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T08:39:38.984 INFO:teuthology.orchestra.run.smithi093.stdout:Relocations : (not relocatable) 2021-11-20T08:39:38.984 INFO:teuthology.orchestra.run.smithi093.stdout:Packager : CentOS BuildSystem 2021-11-20T08:39:38.984 INFO:teuthology.orchestra.run.smithi093.stdout:Vendor : CentOS 2021-11-20T08:39:38.984 INFO:teuthology.orchestra.run.smithi093.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T08:39:38.985 INFO:teuthology.orchestra.run.smithi093.stdout:Summary : Support tools for GRUB. 2021-11-20T08:39:38.985 INFO:teuthology.orchestra.run.smithi093.stdout:Description : 2021-11-20T08:39:38.985 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:39:38.985 INFO:teuthology.orchestra.run.smithi093.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T08:39:38.985 INFO:teuthology.orchestra.run.smithi093.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T08:39:38.986 INFO:teuthology.orchestra.run.smithi093.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T08:39:38.986 INFO:teuthology.orchestra.run.smithi093.stdout:hardware devices. 2021-11-20T08:39:38.986 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:39:38.986 INFO:teuthology.orchestra.run.smithi093.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T08:39:38.987 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T08:39:38.988 INFO:teuthology.task.kernel:Updating grub on smithi093 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:38.988 DEBUG:teuthology.orchestra.run.smithi093:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T08:39:39.127 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T08:39:39.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:39:39.127 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:39:39.128 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:39.128 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:39.128 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:39.128 INFO:teuthology.orchestra.run.smithi102.stdout:Removed: 2021-11-20T08:39:39.129 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T08:39:39.129 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T08:39:39.129 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:39:39.129 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:39:39.259 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel --last | head -n 1 2021-11-20T08:39:39.455 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 08:39:11 AM UTC 2021-11-20T08:39:39.456 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:39.456 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:39.457 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T08:39:39.457 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T08:39:39.457 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2021-11-20T08:39:39.457 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T08:39:39.458 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel --last | head -n 1 2021-11-20T08:39:39.600 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 08:39:11 AM UTC 2021-11-20T08:39:39.601 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:39.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2021-11-20T08:39:39.632 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2021-11-20T08:39:39.632 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2021-11-20T08:39:39.632 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.02 2021-11-20T08:39:39.633 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 99.el8 2021-11-20T08:39:39.633 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2021-11-20T08:39:39.633 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T08:39:39.633 INFO:teuthology.orchestra.run.smithi102.stdout:Group : System Environment/Base 2021-11-20T08:39:39.633 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 9264704 2021-11-20T08:39:39.634 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2021-11-20T08:39:39.634 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T08:39:39.634 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T08:39:39.635 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T08:39:39.635 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T08:39:39.636 INFO:teuthology.orchestra.run.smithi102.stdout:Relocations : (not relocatable) 2021-11-20T08:39:39.636 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : CentOS BuildSystem 2021-11-20T08:39:39.636 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2021-11-20T08:39:39.636 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T08:39:39.637 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2021-11-20T08:39:39.637 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2021-11-20T08:39:39.637 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:39:39.637 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T08:39:39.637 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T08:39:39.638 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T08:39:39.638 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2021-11-20T08:39:39.638 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:39:39.638 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T08:39:39.641 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T08:39:39.642 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:39.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T08:39:40.348 INFO:teuthology.orchestra.run.smithi093.stderr:Generating grub configuration file ... 2021-11-20T08:39:40.953 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2021-11-20T08:39:42.434 INFO:teuthology.orchestra.run.smithi093.stderr:done 2021-11-20T08:39:42.435 DEBUG:teuthology.orchestra.run.smithi093:> mktemp 2021-11-20T08:39:42.477 INFO:teuthology.orchestra.run.smithi093.stdout:/tmp/tmp.jc67fcThfh 2021-11-20T08:39:42.478 DEBUG:teuthology.orchestra.run.smithi093:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jc67fcThfh 2021-11-20T08:39:42.550 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 0666 /tmp/tmp.jc67fcThfh 2021-11-20T08:39:42.706 DEBUG:teuthology.orchestra.remote:smithi093:/tmp/tmp.jc67fcThfh is 6KB 2021-11-20T08:39:42.756 DEBUG:teuthology.orchestra.run.smithi093:> rm -fr /tmp/tmp.jc67fcThfh 2021-11-20T08:39:42.772 DEBUG:teuthology.orchestra.run.smithi093:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T08:39:42.835 INFO:teuthology.orchestra.run.smithi093.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T08:39:42.835 INFO:teuthology.orchestra.run.smithi093.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T08:39:42.835 INFO:teuthology.orchestra.run.smithi093.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T08:39:42.835 INFO:teuthology.orchestra.run.smithi093.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T08:39:42.837 DEBUG:teuthology.orchestra.run.smithi093:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:43.007 DEBUG:teuthology.orchestra.run.smithi093:> sudo shutdown -r now 2021-11-20T08:39:43.139 INFO:teuthology.orchestra.run.smithi102.stderr:done 2021-11-20T08:39:43.141 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2021-11-20T08:39:43.185 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.WR8xkeZXPl 2021-11-20T08:39:43.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WR8xkeZXPl 2021-11-20T08:39:43.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.WR8xkeZXPl 2021-11-20T08:39:43.410 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.WR8xkeZXPl is 6KB 2021-11-20T08:39:43.461 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.WR8xkeZXPl 2021-11-20T08:39:43.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T08:39:43.539 INFO:teuthology.orchestra.run.smithi102.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T08:39:43.539 INFO:teuthology.orchestra.run.smithi102.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T08:39:43.539 INFO:teuthology.orchestra.run.smithi102.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T08:39:43.540 INFO:teuthology.orchestra.run.smithi102.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T08:39:43.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:39:43.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2021-11-20T08:40:13.054 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2021-11-20T08:40:13.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2021-11-20T08:40:13.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:40:13.688 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2021-11-20T08:40:13.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2021-11-20T08:40:13.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:41:13.717 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T08:41:43.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2021-11-20T08:41:43.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:41:43.981 DEBUG:teuthology.orchestra.run.smithi102:> true 2021-11-20T08:41:44.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2021-11-20T08:41:44.227 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T08:41:44.227 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2021-11-20T08:41:44.284 INFO:teuthology.orchestra.run.smithi102.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:41:44.284 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:41:44.285 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T08:41:44.285 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T08:41:45.286 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2021-11-20T08:41:45.287 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T08:41:45.662 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2021-11-20T08:41:45.677 DEBUG:teuthology.parallel:result is None 2021-11-20T08:41:47.568 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T08:42:17.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2021-11-20T08:42:17.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:42:17.947 DEBUG:teuthology.orchestra.run.smithi093:> true 2021-11-20T08:42:18.243 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi093.front.sepia.ceph.com' 2021-11-20T08:42:18.244 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T08:42:18.244 DEBUG:teuthology.orchestra.run.smithi093:> uname -r 2021-11-20T08:42:18.261 INFO:teuthology.orchestra.run.smithi093.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:42:18.263 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T08:42:18.263 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T08:42:18.263 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T08:42:19.264 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2021-11-20T08:42:19.265 DEBUG:teuthology.orchestra.run.smithi093:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T08:42:19.346 INFO:teuthology.orchestra.run.smithi093.stdout:ttyS1 2021-11-20T08:42:19.359 DEBUG:teuthology.parallel:result is None 2021-11-20T08:42:19.359 INFO:teuthology.run_tasks:Running task internal.base... 2021-11-20T08:42:19.418 INFO:teuthology.task.internal:Creating test directory... 2021-11-20T08:42:19.419 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T08:42:19.440 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T08:42:19.490 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-11-20T08:42:19.508 INFO:teuthology.run_tasks:Running task internal.archive... 2021-11-20T08:42:19.554 INFO:teuthology.task.internal:Creating archive directory... 2021-11-20T08:42:19.554 DEBUG:teuthology.orchestra.run.smithi093:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T08:42:19.567 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T08:42:20.155 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-11-20T08:42:20.170 INFO:teuthology.task.internal:Enabling coredump saving... 2021-11-20T08:42:20.170 DEBUG:teuthology.orchestra.run.smithi093:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T08:42:20.181 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T08:42:20.212 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T08:42:20.220 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T08:42:20.223 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T08:42:20.231 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T08:42:20.233 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-11-20T08:42:20.248 INFO:teuthology.task.internal:Configuring sudo... 2021-11-20T08:42:20.249 DEBUG:teuthology.orchestra.run.smithi093:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T08:42:20.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T08:42:20.305 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-11-20T08:42:20.319 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-11-20T08:42:20.320 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T08:42:20.346 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T08:42:20.367 DEBUG:teuthology.orchestra.run.smithi093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T08:42:20.448 DEBUG:teuthology.orchestra.run.smithi093:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T08:42:20.498 DEBUG:teuthology.orchestra.run.smithi093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T08:42:20.581 DEBUG:teuthology.orchestra.run.smithi093:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T08:42:20.626 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:42:20.626 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T08:42:20.693 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T08:42:20.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T08:42:20.791 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T08:42:20.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T08:42:20.917 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:42:20.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T08:42:20.983 DEBUG:teuthology.orchestra.run.smithi093:> sudo service rsyslog restart 2021-11-20T08:42:20.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2021-11-20T08:42:21.042 INFO:teuthology.orchestra.run.smithi093.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T08:42:21.077 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T08:42:21.474 INFO:teuthology.run_tasks:Running task internal.timer... 2021-11-20T08:42:21.483 INFO:teuthology.task.internal:Starting timer... 2021-11-20T08:42:21.483 INFO:teuthology.run_tasks:Running task pcp... 2021-11-20T08:42:21.520 INFO:teuthology.run_tasks:Running task selinux... 2021-11-20T08:42:21.553 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-11-20T08:42:21.554 DEBUG:teuthology.orchestra.run.smithi093:> sudo service auditd rotate 2021-11-20T08:42:21.618 INFO:teuthology.orchestra.run.smithi093.stdout:Rotating logs: [ OK ] 2021-11-20T08:42:21.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2021-11-20T08:42:21.687 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: [ OK ] 2021-11-20T08:42:21.689 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-11-20T08:42:21.689 DEBUG:teuthology.orchestra.run.smithi093:> /usr/sbin/getenforce 2021-11-20T08:42:21.712 INFO:teuthology.orchestra.run.smithi093.stdout:Permissive 2021-11-20T08:42:21.713 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2021-11-20T08:42:21.742 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2021-11-20T08:42:21.742 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi093.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi102.front.sepia.ceph.com': 'permissive'} 2021-11-20T08:42:21.743 DEBUG:teuthology.orchestra.run.smithi093:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T08:42:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T08:42:21.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T08:42:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T08:42:21.808 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-11-20T08:42:21.808 DEBUG:teuthology.orchestra.run.smithi093:> sudo /usr/sbin/setenforce permissive 2021-11-20T08:42:21.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2021-11-20T08:42:21.890 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-11-20T08:42:21.901 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-11-20T08:42:21.902 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-11-20T08:42:21.925 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-11-20T08:42:21.927 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi093.front.sepia.ceph.com,smithi102.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-11-20T08:42:22.315 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-11-20T08:42:23.124 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-11-20T08:42:26.071 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-11-20T08:42:26.179 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:26 +0000 (0:00:00.125) 0:00:00.125 ***** 2021-11-20T08:42:26.191 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:26 +0000 (0:00:00.011) 0:00:00.136 ***** 2021-11-20T08:42:27.617 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi093.front.sepia.ceph.com] 2021-11-20T08:42:27.647 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-11-20T08:42:27.673 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:27 +0000 (0:00:01.482) 0:00:01.618 ***** 2021-11-20T08:42:27.713 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:27 +0000 (0:00:00.039) 0:00:01.657 ***** 2021-11-20T08:42:27.942 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-11-20T08:42:27.943 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T08:42:27.964 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:27 +0000 (0:00:00.251) 0:00:01.909 ***** 2021-11-20T08:42:27.983 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:27 +0000 (0:00:00.018) 0:00:01.928 ***** 2021-11-20T08:42:28.526 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-11-20T08:42:28.528 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [smithi102.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-20T08:42:28.549 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:28 +0000 (0:00:00.566) 0:00:02.495 ***** 2021-11-20T08:42:28.570 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:28 +0000 (0:00:00.019) 0:00:02.514 ***** 2021-11-20T08:42:28.617 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-11-20T08:42:28.619 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:28.639 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:28 +0000 (0:00:00.070) 0:00:02.584 ***** 2021-11-20T08:42:28.646 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:28.669 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:28 +0000 (0:00:00.029) 0:00:02.614 ***** 2021-11-20T08:42:28.904 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-11-20T08:42:28.906 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi093.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:42:28.928 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:28 +0000 (0:00:00.259) 0:00:02.873 ***** 2021-11-20T08:42:28.948 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:28 +0000 (0:00:00.019) 0:00:02.893 ***** 2021-11-20T08:42:29.682 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-11-20T08:42:29.684 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397748.9724698-16785-85276820489009/source", "state": "file", "uid": 0} changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397749.0019717-16788-59481564112347/source", "state": "file", "uid": 0} 2021-11-20T08:42:29.705 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:29 +0000 (0:00:00.756) 0:00:03.650 ***** 2021-11-20T08:42:29.723 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:29 +0000 (0:00:00.018) 0:00:03.668 ***** 2021-11-20T08:42:30.243 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-11-20T08:42:30.245 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T08:42:30.246 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T08:42:30.302 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T08:42:30.304 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T08:42:30.369 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-11-20T08:42:30.370 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T08:42:30.383 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 20 November 2021 08:42:30 +0000 (0:00:00.660) 0:00:04.329 ***** 2021-11-20T08:42:30.904 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] 2021-11-20T08:42:30.923 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-11-20T08:42:30.941 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-11-20T08:42:30.943 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:30 +0000 (0:00:00.557) 0:00:04.886 ***** 2021-11-20T08:42:30.969 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T08:42:30.981 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T08:42:30.986 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T08:42:31.156 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:31 +0000 (0:00:00.215) 0:00:05.101 ***** 2021-11-20T08:42:31.179 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:31 +0000 (0:00:00.022) 0:00:05.124 ***** 2021-11-20T08:42:31.795 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi093.front.sepia.ceph.com] 2021-11-20T08:42:31.827 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-11-20T08:42:31.856 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-11-20T08:42:31.858 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:31 +0000 (0:00:00.678) 0:00:05.802 ***** 2021-11-20T08:42:31.903 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:31 +0000 (0:00:00.045) 0:00:05.848 ***** 2021-11-20T08:42:32.713 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-11-20T08:42:32.715 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.380219", "end": "2021-11-20 08:42:32.691476", "rc": 0, "start": "2021-11-20 08:42:32.311257", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.412801", "end": "2021-11-20 08:42:32.693514", "rc": 0, "start": "2021-11-20 08:42:32.280713", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-20T08:42:32.737 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:32 +0000 (0:00:00.834) 0:00:06.682 ***** 2021-11-20T08:42:32.757 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:32 +0000 (0:00:00.019) 0:00:06.702 ***** 2021-11-20T08:42:32.960 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-11-20T08:42:32.962 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.006010", "end": "2021-11-20 08:42:32.936055", "failed_when_result": false, "rc": 0, "start": "2021-11-20 08:42:32.930045", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010106", "end": "2021-11-20 08:42:32.944318", "failed_when_result": false, "rc": 0, "start": "2021-11-20 08:42:32.934212", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-20T08:42:32.982 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:32 +0000 (0:00:00.225) 0:00:06.928 ***** 2021-11-20T08:42:33.002 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:32 +0000 (0:00:00.020) 0:00:06.948 ***** 2021-11-20T08:42:33.040 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-11-20T08:42:33.042 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.063 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.060) 0:00:07.009 ***** 2021-11-20T08:42:33.071 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.095 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.031) 0:00:07.040 ***** 2021-11-20T08:42:33.133 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-11-20T08:42:33.135 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.156 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.061) 0:00:07.101 ***** 2021-11-20T08:42:33.165 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.191 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.034) 0:00:07.136 ***** 2021-11-20T08:42:33.230 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.252 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.061) 0:00:07.198 ***** 2021-11-20T08:42:33.269 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.292 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.039) 0:00:07.238 ***** 2021-11-20T08:42:33.320 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.342 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.050) 0:00:07.288 ***** 2021-11-20T08:42:33.370 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.394 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.051) 0:00:07.339 ***** 2021-11-20T08:42:33.412 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.041) 0:00:07.381 ***** 2021-11-20T08:42:33.463 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:42:33.486 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.050) 0:00:07.432 ***** 2021-11-20T08:42:33.670 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-11-20T08:42:33.672 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:42:33.693 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.206) 0:00:07.638 ***** 2021-11-20T08:42:33.731 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:42:33.754 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:42:33 +0000 (0:00:00.061) 0:00:07.700 ***** 2021-11-20T08:42:33.980 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T08:42:33.982 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T08:42:34.045 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T08:42:34.172 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T08:42:34.235 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T08:42:34.361 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T08:42:34.435 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T08:42:34.561 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T08:42:34.624 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T08:42:34.758 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T08:42:34.823 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T08:42:34.957 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T08:42:35.021 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T08:42:35.146 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T08:42:35.220 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T08:42:35.347 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T08:42:35.412 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T08:42:35.538 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T08:42:35.613 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T08:42:35.741 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T08:42:35.815 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T08:42:35.931 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T08:42:36.006 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T08:42:36.131 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T08:42:36.195 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T08:42:36.332 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T08:42:36.396 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T08:42:36.523 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T08:42:36.585 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T08:42:36.721 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T08:42:36.784 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T08:42:36.911 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T08:42:36.975 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T08:42:37.111 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T08:42:37.174 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T08:42:37.310 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T08:42:37.375 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T08:42:37.500 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T08:42:37.566 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T08:42:37.705 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T08:42:37.801 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T08:42:37.926 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T08:42:38.044 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T08:42:38.119 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T08:42:38.278 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T08:42:38.325 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T08:42:38.501 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T08:42:38.523 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T08:42:38.703 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T08:42:38.736 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T08:42:38.904 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T08:42:38.939 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T08:42:39.107 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T08:42:39.119 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T08:42:39.301 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T08:42:39.323 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T08:42:39.501 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T08:42:39.524 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T08:42:39.699 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T08:42:39.721 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T08:42:39.888 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T08:42:39.910 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T08:42:40.089 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T08:42:40.110 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T08:42:40.291 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T08:42:40.313 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T08:42:40.479 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T08:42:40.501 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T08:42:40.682 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T08:42:40.704 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T08:42:40.883 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T08:42:40.905 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T08:42:41.074 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T08:42:41.096 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-20T08:42:41.264 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-20T08:42:41.298 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T08:42:41.463 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T08:42:41.496 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T08:42:41.674 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T08:42:41.686 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T08:42:41.874 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T08:42:41.888 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-20T08:42:42.065 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-20T08:42:42.097 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-20T08:42:42.254 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-20T08:42:42.296 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T08:42:42.445 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T08:42:42.501 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-20T08:42:42.648 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-20T08:42:42.701 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T08:42:42.847 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T08:42:42.890 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T08:42:43.048 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T08:42:43.092 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T08:42:43.250 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T08:42:43.291 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T08:42:43.447 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T08:42:43.490 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T08:42:43.647 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T08:42:43.690 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T08:42:43.837 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T08:42:43.891 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T08:42:44.041 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T08:42:44.094 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-20T08:42:44.240 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-20T08:42:44.294 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T08:42:44.443 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T08:42:44.486 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T08:42:44.633 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T08:42:44.686 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T08:42:44.837 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T08:42:44.892 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T08:42:45.029 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T08:42:45.092 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T08:42:45.229 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T08:42:45.292 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T08:42:45.427 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T08:42:45.480 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T08:42:45.615 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T08:42:45.681 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T08:42:45.817 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T08:42:45.881 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T08:42:46.017 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T08:42:46.070 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T08:42:46.217 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T08:42:46.271 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T08:42:46.409 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T08:42:46.460 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T08:42:46.606 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T08:42:46.661 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T08:42:46.808 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T08:42:46.862 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T08:42:46.999 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T08:42:47.055 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T08:42:47.193 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T08:42:47.257 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T08:42:47.393 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T08:42:47.448 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T08:42:47.642 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T08:42:47.712 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T08:42:47.788 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T08:42:47.928 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T08:42:47.993 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T08:42:48.129 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T08:42:48.204 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T08:42:48.331 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T08:42:48.396 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T08:42:48.545 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T08:42:48.599 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T08:42:48.728 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T08:42:48.802 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T08:42:48.929 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T08:42:49.003 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T08:42:49.119 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T08:42:49.194 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T08:42:49.320 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T08:42:49.395 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T08:42:49.521 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T08:42:49.596 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T08:42:49.716 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T08:42:49.791 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T08:42:49.917 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T08:42:49.991 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T08:42:50.117 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T08:42:50.191 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T08:42:50.317 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T08:42:50.382 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T08:42:50.511 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T08:42:50.573 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T08:42:50.712 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T08:42:50.775 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T08:42:50.912 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T08:42:50.965 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T08:42:51.105 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T08:42:51.168 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T08:42:51.314 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T08:42:51.398 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T08:42:51.560 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T08:42:51.626 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T08:42:51.786 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T08:42:51.865 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T08:42:52.023 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T08:42:52.066 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T08:42:52.235 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T08:42:52.268 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T08:42:52.439 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T08:42:52.483 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T08:42:52.629 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T08:42:52.672 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T08:42:52.832 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T08:42:52.875 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T08:42:53.022 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T08:42:53.075 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T08:42:53.224 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T08:42:53.277 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T08:42:53.425 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T08:42:53.468 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T08:42:53.625 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T08:42:53.668 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T08:42:53.816 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T08:42:53.870 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T08:42:54.019 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T08:42:54.062 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T08:42:54.211 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T08:42:54.264 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T08:42:54.413 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T08:42:54.456 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T08:42:54.602 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T08:42:54.656 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T08:42:54.807 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T08:42:54.849 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T08:42:54.996 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T08:42:55.052 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T08:42:55.189 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T08:42:55.253 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T08:42:55.390 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T08:42:55.443 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T08:42:55.581 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T08:42:55.633 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T08:42:55.783 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T08:42:55.836 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T08:42:55.972 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T08:42:56.025 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T08:42:56.172 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T08:42:56.225 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T08:42:56.373 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T08:42:56.427 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T08:42:56.574 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T08:42:56.627 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T08:42:56.774 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T08:42:56.828 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T08:42:56.975 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T08:42:57.018 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T08:42:57.165 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T08:42:57.217 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T08:42:57.365 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T08:42:57.419 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T08:42:57.566 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T08:42:57.622 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T08:42:57.761 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T08:42:57.820 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T08:42:57.968 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T08:42:58.021 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T08:42:58.168 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T08:42:58.223 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T08:42:58.361 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T08:42:58.425 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T08:42:58.551 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T08:42:58.627 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T08:42:58.753 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T08:42:58.817 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T08:42:58.954 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T08:42:59.018 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T08:42:59.165 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T08:42:59.212 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T08:42:59.359 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T08:42:59.413 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T08:42:59.559 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T08:42:59.622 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T08:42:59.761 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T08:42:59.835 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T08:42:59.950 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T08:43:00.023 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T08:43:00.150 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T08:43:00.225 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T08:43:00.350 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T08:43:00.425 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T08:43:01.894 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T08:43:01.948 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T08:43:03.759 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T08:43:03.760 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T08:43:05.266 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T08:43:05.392 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T08:43:06.794 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T08:43:06.868 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T08:43:08.331 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T08:43:08.364 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T08:43:09.834 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T08:43:09.866 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T08:43:11.371 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T08:43:11.383 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T08:43:12.936 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T08:43:12.947 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T08:43:14.467 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T08:43:14.510 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T08:43:16.005 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T08:43:16.091 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T08:43:17.579 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T08:43:17.644 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T08:43:19.181 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T08:43:19.256 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T08:43:20.720 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T08:43:21.055 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T08:43:22.312 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T08:43:22.732 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T08:43:23.968 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T08:43:24.387 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T08:43:25.547 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T08:43:26.018 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T08:43:27.122 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T08:43:27.705 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T08:43:28.705 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T08:43:29.351 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T08:43:30.313 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T08:43:31.002 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T08:43:31.913 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T08:43:32.659 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T08:43:33.510 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T08:43:33.632 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:33 +0000 (0:00:59.878) 0:01:07.578 ***** 2021-11-20T08:43:33.701 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-20T08:43:33.722 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:33 +0000 (0:00:00.089) 0:01:07.668 ***** 2021-11-20T08:43:33.813 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-20T08:43:33.815 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-20T08:43:33.817 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-20T08:43:33.818 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-20T08:43:33.820 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-20T08:43:33.822 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-20T08:43:33.823 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-20T08:43:33.825 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a9 2021-11-20T08:43:33.826 INFO:teuthology.task.ansible.out:5579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-20T08:43:33.827 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-20T08:43:33.829 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-20T08:43:33.830 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-20T08:43:33.832 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-20T08:43:33.834 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-20T08:43:33.835 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5 2021-11-20T08:43:33.836 INFO:teuthology.task.ansible.out:299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-20T08:43:33.837 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-20T08:43:33.862 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:33 +0000 (0:00:00.139) 0:01:07.807 ***** 2021-11-20T08:43:34.350 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T08:43:34.352 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T08:43:34.473 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:34 +0000 (0:00:00.610) 0:01:08.418 ***** 2021-11-20T08:43:34.534 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-20T08:43:34.558 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:34 +0000 (0:00:00.085) 0:01:08.504 ***** 2021-11-20T08:43:34.648 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-20T08:43:34.650 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-20T08:43:34.652 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-20T08:43:34.653 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-20T08:43:34.655 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-20T08:43:34.657 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-20T08:43:34.658 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-20T08:43:34.660 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-20T08:43:34.661 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-20T08:43:34.663 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-20T08:43:34.665 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-20T08:43:34.666 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-20T08:43:34.668 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-20T08:43:34.670 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-20T08:43:34.671 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f2283 2021-11-20T08:43:34.672 INFO:teuthology.task.ansible.out:5b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-20T08:43:34.699 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:34 +0000 (0:00:00.140) 0:01:08.644 ***** 2021-11-20T08:43:34.877 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-20T08:43:34.880 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T08:43:34.906 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:34 +0000 (0:00:00.207) 0:01:08.851 ***** 2021-11-20T08:43:35.253 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T08:43:35.255 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T08:43:35.358 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-20T08:43:35.360 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T08:43:35.380 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:43:35 +0000 (0:00:00.474) 0:01:09.326 ***** 2021-11-20T08:43:35.460 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T08:43:35.462 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T08:43:35.796 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T08:43:35.859 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T08:43:36.024 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T08:43:36.026 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T08:43:36.067 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T08:43:36.234 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T08:43:36.276 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T08:43:36.442 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-20T08:43:36.444 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T08:43:36.495 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T08:43:36.497 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T08:43:36.653 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T08:43:36.706 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T08:43:36.865 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T08:43:36.918 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-20T08:43:36.919 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T08:43:37.074 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T08:43:37.086 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:43:37.130 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T08:43:37.287 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T08:43:37.289 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T08:43:37.329 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T08:43:37.506 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T08:43:37.508 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T08:43:37.517 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:43:37.528 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:43:37.542 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T08:43:37.553 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:43:37.730 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T08:43:37.754 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T08:43:37.757 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T08:43:37.943 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T08:43:37.944 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-20T08:43:37.976 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T08:43:37.978 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T08:43:37.986 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:43:37.997 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:43:38.155 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T08:43:38.208 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T08:43:38.367 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T08:43:38.431 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T08:43:38.432 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-20T08:43:38.588 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T08:43:38.652 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T08:43:38.798 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T08:43:38.800 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T08:43:38.863 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T08:43:38.998 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T08:43:39.000 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T08:43:39.082 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T08:43:39.209 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T08:43:39.303 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T08:43:39.305 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T08:43:39.418 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T08:43:39.513 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T08:43:39.514 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T08:43:39.628 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T08:43:39.629 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T08:43:39.721 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T08:43:39.836 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T08:43:39.931 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T08:43:40.035 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T08:43:40.223 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T08:43:40.225 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T08:43:40.245 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T08:43:40.433 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T08:43:40.454 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T08:43:40.643 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T08:43:40.665 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T08:43:40.853 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T08:43:40.875 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T08:43:40.876 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T08:43:41.062 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T08:43:41.084 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T08:43:41.271 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T08:43:41.293 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T08:43:41.481 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T08:43:41.483 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T08:43:41.492 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T08:43:41.690 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T08:43:41.705 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T08:43:41.899 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T08:43:41.910 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T08:43:42.100 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T08:43:42.111 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T08:43:42.309 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T08:43:42.330 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T08:43:42.517 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T08:43:42.529 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T08:43:42.531 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T08:43:42.717 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T08:43:42.739 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T08:43:42.742 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T08:43:42.927 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T08:43:42.950 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T08:43:43.147 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T08:43:43.149 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T08:43:43.159 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T08:43:43.347 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T08:43:43.349 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T08:43:43.358 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T08:43:43.557 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T08:43:43.569 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T08:43:43.571 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T08:43:43.767 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T08:43:43.788 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T08:43:43.790 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T08:43:43.792 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T08:43:43.965 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T08:43:44.008 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T08:43:44.175 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T08:43:44.177 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T08:43:44.217 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T08:43:44.383 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T08:43:44.385 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T08:43:44.386 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T08:43:44.415 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T08:43:44.592 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T08:43:44.624 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T08:43:44.795 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T08:43:44.828 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T08:43:45.005 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T08:43:45.037 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T08:43:45.039 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T08:43:45.215 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T08:43:45.247 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T08:43:45.412 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T08:43:45.444 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T08:43:45.621 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T08:43:45.623 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T08:43:45.664 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T08:43:45.665 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T08:43:45.830 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T08:43:45.862 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T08:43:46.039 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T08:43:46.072 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T08:43:46.243 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T08:43:46.245 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T08:43:46.275 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T08:43:46.277 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T08:43:46.451 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T08:43:46.483 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T08:43:46.660 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T08:43:46.682 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T08:43:46.868 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T08:43:46.870 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T08:43:46.900 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T08:43:47.067 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T08:43:47.109 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T08:43:47.277 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T08:43:47.309 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T08:43:47.486 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T08:43:47.528 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-20T08:43:47.684 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T08:43:47.737 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-20T08:43:47.893 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T08:43:47.946 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T08:43:48.094 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-20T08:43:48.356 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "valida 2021-11-20T08:43:48.359 INFO:teuthology.task.ansible.out:te_certs": true} 2021-11-20T08:43:48.389 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T08:43:48.579 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T08:43:48.611 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T08:43:48.788 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-20T08:43:48.830 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T08:43:48.996 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T08:43:49.028 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T08:43:49.195 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T08:43:49.237 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T08:43:49.403 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T08:43:49.456 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T08:43:49.603 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T08:43:49.666 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T08:43:49.813 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T08:43:49.877 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T08:43:50.034 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T08:43:50.097 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T08:43:50.239 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T08:43:50.302 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T08:43:50.447 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T08:43:50.511 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T08:43:50.668 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T08:43:50.720 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T08:43:50.722 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T08:43:50.877 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T08:43:50.920 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T08:43:51.076 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T08:43:51.128 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T08:43:51.285 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T08:43:51.287 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T08:43:51.327 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T08:43:51.493 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T08:43:51.536 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T08:43:51.537 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T08:43:51.703 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T08:43:51.755 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T08:43:51.911 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T08:43:51.954 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T08:43:52.120 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T08:43:52.123 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T08:43:52.174 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T08:43:52.333 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T08:43:52.385 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T08:43:52.541 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T08:43:52.594 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T08:43:52.751 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T08:43:52.793 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T08:43:52.971 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T08:43:53.003 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T08:43:53.181 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T08:43:53.214 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T08:43:53.216 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T08:43:53.427 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T08:43:53.430 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T08:43:53.662 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T08:43:53.701 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T08:43:53.866 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T08:43:53.929 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T08:43:53.931 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T08:43:54.075 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T08:43:54.158 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T08:43:54.296 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-20T08:43:54.380 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T08:43:54.494 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T08:43:54.588 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T08:43:54.704 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T08:43:54.705 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T08:43:54.787 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T08:43:54.913 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-20T08:43:54.998 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-20T08:43:55.114 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T08:43:55.198 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T08:43:55.323 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-20T08:43:55.407 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T08:43:55.408 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T08:43:55.521 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T08:43:55.615 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-20T08:43:55.731 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-20T08:43:55.816 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T08:43:55.941 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T08:43:56.015 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-20T08:43:56.150 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T08:43:56.224 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T08:43:56.360 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T08:43:56.435 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-20T08:43:56.560 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T08:43:56.643 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T08:43:56.769 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T08:43:56.856 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T08:43:56.970 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T08:43:57.065 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T08:43:57.179 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T08:43:57.273 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T08:43:57.388 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-20T08:43:57.482 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T08:43:57.596 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T08:43:57.680 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T08:43:57.795 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T08:43:57.899 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T08:43:58.003 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T08:43:58.108 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-20T08:43:58.213 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T08:43:58.308 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T08:43:58.413 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T08:43:58.517 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T08:43:58.623 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T08:43:58.625 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T08:43:58.728 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T08:43:58.833 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T08:43:58.928 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T08:43:59.034 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T08:43:59.139 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T08:43:59.234 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T08:43:59.349 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T08:43:59.351 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T08:43:59.444 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T08:43:59.549 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T08:43:59.665 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T08:43:59.759 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T08:43:59.864 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T08:43:59.958 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T08:44:00.073 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T08:44:00.167 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T08:44:00.282 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T08:44:00.379 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T08:44:00.484 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-20T08:44:00.578 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T08:44:00.694 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T08:44:00.788 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T08:44:00.902 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T08:44:00.996 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T08:44:01.110 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T08:44:01.204 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-20T08:44:01.319 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T08:44:01.321 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T08:44:01.413 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T08:44:01.529 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T08:44:01.624 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T08:44:01.730 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T08:44:01.824 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T08:44:01.939 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T08:44:01.941 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-20T08:44:02.035 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T08:44:02.037 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T08:44:02.150 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T08:44:02.234 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T08:44:02.351 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T08:44:02.445 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T08:44:02.561 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T08:44:02.666 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T08:44:02.668 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-20T08:44:02.771 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T08:44:02.866 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T08:44:02.982 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T08:44:03.070 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T08:44:03.185 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T08:44:03.278 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T08:44:03.405 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T08:44:03.489 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T08:44:03.614 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T08:44:03.700 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T08:44:03.816 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T08:44:03.900 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T08:44:04.026 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T08:44:04.114 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T08:44:04.239 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-20T08:44:04.335 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T08:44:04.449 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-20T08:44:04.544 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T08:44:04.659 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-20T08:44:04.753 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T08:44:04.867 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-20T08:44:04.962 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-20T08:44:05.067 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-20T08:44:05.171 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-20T08:44:05.318 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T08:44:05.381 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-20T08:44:05.529 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T08:44:05.592 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-20T08:44:05.749 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T08:44:05.865 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-20T08:44:05.969 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T08:44:06.084 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T08:44:06.179 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T08:44:06.294 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T08:44:06.393 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T08:44:06.534 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T08:44:06.605 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T08:44:06.776 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T08:44:06.820 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T08:44:07.013 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T08:44:07.034 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:07 +0000 (0:00:31.653) 0:01:40.979 ***** 2021-11-20T08:44:07.137 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T08:44:07.139 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.149 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.160 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.162 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.171 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.182 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.184 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.194 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.196 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T08:44:07.404 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T08:44:07.415 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T08:44:07.417 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-11-20T08:44:07.419 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.427 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.614 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T08:44:07.616 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T08:44:07.688 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T08:44:07.690 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-20T08:44:07.712 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:07 +0000 (0:00:00.676) 0:01:41.656 ***** 2021-11-20T08:44:07.814 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.825 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.836 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.837 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-20T08:44:07.838 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072ac 2021-11-20T08:44:07.840 INFO:teuthology.task.ansible.out:b3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.848 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.850 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.860 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.862 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.873 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.874 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.886 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.888 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.898 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.900 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.909 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.911 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.921 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.932 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.933 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.944 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.955 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.957 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.967 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.978 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.980 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.989 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:07.991 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.001 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.012 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.014 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.026 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.036 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.047 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.049 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.058 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.070 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.072 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.081 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.092 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-20T08:44:08.094 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.105 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.107 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.117 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.129 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.140 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.152 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.163 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.165 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.175 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.186 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.197 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.199 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.210 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.221 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.223 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.233 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.244 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.246 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.256 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.267 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.269 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.280 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.292 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.293 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.305 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.307 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.317 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.319 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-20T08:44:08.320 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.332 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.332 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.344 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.355 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.367 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.369 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.378 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.380 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.390 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.401 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.403 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.413 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.414 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.425 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.436 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.438 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.448 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.449 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.459 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.470 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.472 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.482 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.493 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.495 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.505 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.506 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.516 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.517 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-20T08:44:08.519 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.529 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.530 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.540 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.542 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"a 2021-11-20T08:44:08.544 INFO:teuthology.task.ansible.out:nsible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.553 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.554 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.565 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.566 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.576 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.578 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.589 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.590 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.601 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.601 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.612 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.614 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.624 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.626 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.637 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.639 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.649 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.651 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.662 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.663 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.673 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.674 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.684 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.686 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.697 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.699 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.708 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.710 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.721 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.722 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.733 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.735 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.745 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.747 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.758 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.769 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.770 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.781 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.782 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.793 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.795 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.805 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.806 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.808 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.818 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.820 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.830 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.832 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.842 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.844 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.855 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.857 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.867 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.868 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.878 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.879 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.880 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.901 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.903 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.925 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:08 +0000 (0:00:01.214) 0:01:42.871 ***** 2021-11-20T08:44:08.933 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.934 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.934 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.936 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.948 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.949 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.959 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.960 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.971 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.982 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.983 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.983 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.995 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:08.996 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.018 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:09 +0000 (0:00:00.092) 0:01:42.963 ***** 2021-11-20T08:44:09.025 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T08:44:09.027 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.027 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.029 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.040 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.041 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.051 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.053 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.063 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.074 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.076 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.086 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.087 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.097 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.109 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.110 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.120 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.121 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.131 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.133 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.143 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.154 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.156 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.165 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.167 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.177 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.188 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.189 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.199 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.201 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.211 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.212 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.212 INFO:teuthology.task.ansible.out: 2021-11-20T08:44:09.223 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.234 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.236 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.247 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.257 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.269 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.280 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.281 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.291 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.302 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.304 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi102.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.314 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.316 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.326 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.327 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.337 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.339 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.349 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.360 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.362 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.371 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.373 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.383 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.384 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.394 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.405 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.406 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.416 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.418 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.428 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.429 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.439 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.450 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.451 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.461 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.463 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.474 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.494 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T08:44:09.496 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T08:44:09.516 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:09 +0000 (0:00:00.498) 0:01:43.462 ***** 2021-11-20T08:44:09.587 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:09.608 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:09 +0000 (0:00:00.092) 0:01:43.554 ***** 2021-11-20T08:44:09.678 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T08:44:09.679 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T08:44:09.879 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T08:44:09.890 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-20T08:44:10.067 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T08:44:10.079 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T08:44:10.258 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T08:44:10.269 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T08:44:10.456 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T08:44:10.457 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T08:44:10.644 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T08:44:10.655 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T08:44:10.842 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-20T08:44:10.854 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T08:44:11.040 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T08:44:11.042 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T08:44:11.230 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T08:44:11.240 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T08:44:11.419 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-20T08:44:11.430 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-20T08:44:11.617 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T08:44:11.629 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T08:44:11.810 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-20T08:44:11.821 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T08:44:12.018 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T08:44:12.050 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-20T08:44:12.226 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-20T08:44:12.248 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T08:44:12.426 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T08:44:12.449 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-20T08:44:12.630 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T08:44:12.653 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T08:44:12.833 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T08:44:12.855 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-20T08:44:13.021 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T08:44:13.053 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T08:44:13.219 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T08:44:13.251 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T08:44:13.408 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T08:44:13.440 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T08:44:13.608 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T08:44:13.639 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T08:44:13.796 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T08:44:13.828 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T08:44:13.994 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T08:44:14.026 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T08:44:14.182 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T08:44:14.215 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T08:44:14.384 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T08:44:14.417 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T08:44:14.574 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T08:44:14.617 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T08:44:14.765 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T08:44:14.810 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T08:44:14.958 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T08:44:15.022 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T08:44:15.149 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T08:44:15.212 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T08:44:15.338 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T08:44:15.402 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T08:44:15.538 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T08:44:15.602 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T08:44:15.738 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T08:44:15.792 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T08:44:15.937 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T08:44:15.990 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T08:44:16.125 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T08:44:16.188 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T08:44:16.315 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T08:44:16.378 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T08:44:16.504 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T08:44:16.578 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T08:44:16.694 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T08:44:16.767 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T08:44:16.958 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T08:44:17.154 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T08:44:17.343 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T08:44:18.150 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T08:44:18.917 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T08:44:19.683 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T08:44:20.471 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T08:44:21.230 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T08:44:22.061 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T08:44:22.747 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T08:44:23.643 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T08:44:24.287 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T08:44:25.182 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T08:44:25.755 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T08:44:25.964 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T08:44:26.175 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T08:44:26.525 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T08:44:26.596 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T08:44:26.742 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T08:44:26.785 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T08:44:26.951 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T08:44:26.984 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T08:44:27.151 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T08:44:27.173 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T08:44:27.341 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T08:44:27.373 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T08:44:27.533 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T08:44:27.556 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T08:44:27.735 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T08:44:27.757 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T08:44:27.829 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:27 +0000 (0:00:18.220) 0:02:01.775 ***** 2021-11-20T08:44:27.935 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T08:44:28.123 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T08:44:28.314 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T08:44:28.408 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-20T08:44:28.410 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397867.9413135-24999-97221090591734/source", "state": "file", "uid": 0} 2021-11-20T08:44:28.430 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:28 +0000 (0:00:00.600) 0:02:02.376 ***** 2021-11-20T08:44:28.499 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi102.front.sepia.ceph.com] => { "msg": "Host smithi102.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T08:44:28.522 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:28 +0000 (0:00:00.091) 0:02:02.467 ***** 2021-11-20T08:44:28.528 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T08:44:28.530 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T08:44:28.603 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-20T08:44:28.605 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T08:44:28.626 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:28 +0000 (0:00:00.104) 0:02:02.572 ***** 2021-11-20T08:44:28.693 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T08:44:28.695 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:28.722 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:28 +0000 (0:00:00.095) 0:02:02.667 ***** 2021-11-20T08:44:28.729 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T08:44:28.731 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T08:44:28.789 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:28 +0000 (0:00:00.066) 0:02:02.734 ***** 2021-11-20T08:44:29.045 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-20T08:44:29.047 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.079599", "end": "2021-11-20 08:44:29.020215", "rc": 0, "start": "2021-11-20 08:44:28.940616", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T08:44:29.068 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.279) 0:02:03.014 ***** 2021-11-20T08:44:29.138 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-11-20T08:44:29.140 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.163 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.094) 0:02:03.108 ***** 2021-11-20T08:44:29.233 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T08:44:29.235 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.255 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.091) 0:02:03.200 ***** 2021-11-20T08:44:29.325 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-11-20T08:44:29.327 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.349 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.093) 0:02:03.294 ***** 2021-11-20T08:44:29.356 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-20T08:44:29.357 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397868.8850374-25074-134529319225046/source", "state": "file", "uid": 0} 2021-11-20T08:44:29.385 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.036) 0:02:03.330 ***** 2021-11-20T08:44:29.413 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-20T08:44:29.415 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.438 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.053) 0:02:03.384 ***** 2021-11-20T08:44:29.455 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-20T08:44:29.457 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => { "msg": "Host smithi093.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T08:44:29.478 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.040) 0:02:03.424 ***** 2021-11-20T08:44:29.506 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-20T08:44:29.508 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.529 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.050) 0:02:03.475 ***** 2021-11-20T08:44:29.557 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-20T08:44:29.558 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T08:44:29.580 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.050) 0:02:03.525 ***** 2021-11-20T08:44:29.607 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T08:44:29.609 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.632 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.051) 0:02:03.577 ***** 2021-11-20T08:44:29.650 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T08:44:29.652 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.674 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.041) 0:02:03.619 ***** 2021-11-20T08:44:29.713 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-11-20T08:44:29.715 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:29.737 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.063) 0:02:03.683 ***** 2021-11-20T08:44:29.823 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-11-20T08:44:29.825 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.847 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.109) 0:02:03.792 ***** 2021-11-20T08:44:29.917 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-20T08:44:29.919 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:29.940 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:29 +0000 (0:00:00.093) 0:02:03.885 ***** 2021-11-20T08:44:30.002 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-20T08:44:30.004 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.083548", "end": "2021-11-20 08:44:29.981592", "rc": 0, "start": "2021-11-20 08:44:29.898044", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T08:44:30.034 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.093) 0:02:03.979 ***** 2021-11-20T08:44:30.043 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-20T08:44:30.045 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.071 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.035) 0:02:04.014 ***** 2021-11-20T08:44:30.118 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-11-20T08:44:30.120 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.142 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.073) 0:02:04.088 ***** 2021-11-20T08:44:30.161 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T08:44:30.163 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:30.185 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.042) 0:02:04.130 ***** 2021-11-20T08:44:30.222 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T08:44:30.224 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.245 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.060) 0:02:04.190 ***** 2021-11-20T08:44:30.262 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-20T08:44:30.264 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.285 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.040) 0:02:04.230 ***** 2021-11-20T08:44:30.322 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-11-20T08:44:30.324 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.063) 0:02:04.293 ***** 2021-11-20T08:44:30.365 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T08:44:30.368 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:30.389 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.041) 0:02:04.335 ***** 2021-11-20T08:44:30.418 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-20T08:44:30.420 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.446 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.056) 0:02:04.391 ***** 2021-11-20T08:44:30.463 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-11-20T08:44:30.465 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.490 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.041) 0:02:04.433 ***** 2021-11-20T08:44:30.517 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-20T08:44:30.519 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.541 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.053) 0:02:04.487 ***** 2021-11-20T08:44:30.559 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-11-20T08:44:30.561 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.582 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.041) 0:02:04.528 ***** 2021-11-20T08:44:30.610 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T08:44:30.612 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.636 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.053) 0:02:04.581 ***** 2021-11-20T08:44:30.654 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-11-20T08:44:30.656 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.679 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.043) 0:02:04.624 ***** 2021-11-20T08:44:30.719 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-11-20T08:44:30.720 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi093.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:30.743 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.063) 0:02:04.688 ***** 2021-11-20T08:44:30.750 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-11-20T08:44:30.751 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.774 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.031) 0:02:04.720 ***** 2021-11-20T08:44:30.833 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-11-20T08:44:30.835 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.855 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.080) 0:02:04.801 ***** 2021-11-20T08:44:30.862 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-20T08:44:30.864 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:30.887 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.031) 0:02:04.832 ***** 2021-11-20T08:44:30.927 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-20T08:44:30.929 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.951 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.063) 0:02:04.896 ***** 2021-11-20T08:44:30.959 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T08:44:30.960 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:30.983 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:30 +0000 (0:00:00.031) 0:02:04.928 ***** 2021-11-20T08:44:31.031 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-20T08:44:31.032 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.054 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.071) 0:02:04.999 ***** 2021-11-20T08:44:31.060 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-11-20T08:44:31.062 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.085 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.030) 0:02:05.030 ***** 2021-11-20T08:44:31.122 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T08:44:31.124 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:31.147 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.062) 0:02:05.092 ***** 2021-11-20T08:44:31.165 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-20T08:44:31.166 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.189 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.041) 0:02:05.134 ***** 2021-11-20T08:44:31.237 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-20T08:44:31.239 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.261 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.071) 0:02:05.206 ***** 2021-11-20T08:44:31.267 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-20T08:44:31.269 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.291 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.029) 0:02:05.236 ***** 2021-11-20T08:44:31.329 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T08:44:31.331 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:31.353 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.062) 0:02:05.299 ***** 2021-11-20T08:44:31.374 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-20T08:44:31.375 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.398 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.044) 0:02:05.344 ***** 2021-11-20T08:44:31.427 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-11-20T08:44:31.429 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.451 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.053) 0:02:05.397 ***** 2021-11-20T08:44:31.471 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-20T08:44:31.472 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.494 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.042) 0:02:05.440 ***** 2021-11-20T08:44:31.525 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.559 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.062) 0:02:05.503 ***** 2021-11-20T08:44:31.600 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-20T08:44:31.602 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:31.624 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.066) 0:02:05.569 ***** 2021-11-20T08:44:31.631 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-11-20T08:44:31.632 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.658 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.034) 0:02:05.604 ***** 2021-11-20T08:44:31.688 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-20T08:44:31.689 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.720 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.061) 0:02:05.665 ***** 2021-11-20T08:44:31.730 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-11-20T08:44:31.732 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.754 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.034) 0:02:05.700 ***** 2021-11-20T08:44:31.793 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-20T08:44:31.795 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.817 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.062) 0:02:05.762 ***** 2021-11-20T08:44:31.834 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-20T08:44:31.836 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:31.860 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.043) 0:02:05.806 ***** 2021-11-20T08:44:31.931 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:31.953 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:31 +0000 (0:00:00.093) 0:02:05.899 ***** 2021-11-20T08:44:32.022 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-11-20T08:44:32.024 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:32.044 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.090) 0:02:05.989 ***** 2021-11-20T08:44:32.093 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:44:32.115 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.071) 0:02:06.060 ***** 2021-11-20T08:44:32.117 INFO:teuthology.task.ansible.out: 2021-11-20T08:44:32.121 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-20T08:44:32.123 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:32.147 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.031) 0:02:06.092 ***** 2021-11-20T08:44:32.149 INFO:teuthology.task.ansible.out: 2021-11-20T08:44:32.217 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-20T08:44:32.219 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:32.239 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.091) 0:02:06.184 ***** 2021-11-20T08:44:32.308 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-20T08:44:32.310 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:32.332 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.093) 0:02:06.277 ***** 2021-11-20T08:44:32.402 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-20T08:44:32.404 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:32.433 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.100) 0:02:06.378 ***** 2021-11-20T08:44:32.503 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-20T08:44:32.505 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T08:44:32.535 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.097) 0:02:06.476 ***** 2021-11-20T08:44:32.602 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:32.623 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.092) 0:02:06.569 ***** 2021-11-20T08:44:32.715 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-20T08:44:32.717 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:32.740 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:32 +0000 (0:00:00.116) 0:02:06.686 ***** 2021-11-20T08:44:32.883 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-20T08:44:32.885 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-20T08:44:33.021 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-11-20T08:44:33.023 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:44:33.044 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:33 +0000 (0:00:00.304) 0:02:06.990 ***** 2021-11-20T08:44:33.338 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T08:44:33.362 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:33 +0000 (0:00:00.317) 0:02:07.307 ***** 2021-11-20T08:44:33.584 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-20T08:44:33.868 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T08:44:33.870 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T08:44:34.037 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-20T08:44:34.039 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T08:44:34.062 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:34 +0000 (0:00:00.699) 0:02:08.007 ***** 2021-11-20T08:44:34.329 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T08:44:34.331 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T08:44:34.354 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:34 +0000 (0:00:00.292) 0:02:08.299 ***** 2021-11-20T08:44:34.433 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T08:44:34.435 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:34.456 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:34 +0000 (0:00:00.102) 0:02:08.402 ***** 2021-11-20T08:44:34.526 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:34.557 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:34 +0000 (0:00:00.100) 0:02:08.502 ***** 2021-11-20T08:44:34.615 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T08:44:34.617 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T08:44:34.627 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:34.650 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:34 +0000 (0:00:00.093) 0:02:08.595 ***** 2021-11-20T08:44:34.730 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:34.752 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:34 +0000 (0:00:00.101) 0:02:08.697 ***** 2021-11-20T08:44:34.853 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-11-20T08:44:34.855 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:34.865 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:34.867 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:34.905 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:34 +0000 (0:00:00.153) 0:02:08.851 ***** 2021-11-20T08:44:34.987 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T08:44:34.989 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.014 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.108) 0:02:08.960 ***** 2021-11-20T08:44:35.064 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T08:44:35.065 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T08:44:35.090 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.075) 0:02:09.035 ***** 2021-11-20T08:44:35.100 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-20T08:44:35.102 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.127 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.035) 0:02:09.071 ***** 2021-11-20T08:44:35.164 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T08:44:35.166 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.191 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.066) 0:02:09.137 ***** 2021-11-20T08:44:35.210 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-11-20T08:44:35.213 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.237 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.045) 0:02:09.182 ***** 2021-11-20T08:44:35.264 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T08:44:35.266 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.288 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.051) 0:02:09.233 ***** 2021-11-20T08:44:35.326 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-11-20T08:44:35.328 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.353 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.065) 0:02:09.299 ***** 2021-11-20T08:44:35.371 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-11-20T08:44:35.373 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.398 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.045) 0:02:09.344 ***** 2021-11-20T08:44:35.419 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-11-20T08:44:35.420 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.446 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.046) 0:02:09.391 ***** 2021-11-20T08:44:35.463 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-11-20T08:44:35.465 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.488 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.042) 0:02:09.434 ***** 2021-11-20T08:44:35.559 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi093.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.571 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.572 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.595 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.106) 0:02:09.540 ***** 2021-11-20T08:44:35.668 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T08:44:35.669 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.691 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.095) 0:02:09.636 ***** 2021-11-20T08:44:35.770 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-20T08:44:35.772 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.792 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.101) 0:02:09.737 ***** 2021-11-20T08:44:35.862 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-11-20T08:44:35.865 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.892 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.099) 0:02:09.837 ***** 2021-11-20T08:44:35.971 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:35.993 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:35 +0000 (0:00:00.100) 0:02:09.938 ***** 2021-11-20T08:44:36.071 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-11-20T08:44:36.073 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:36.096 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:36 +0000 (0:00:00.103) 0:02:10.042 ***** 2021-11-20T08:44:36.520 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-11-20T08:44:36.523 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.223051", "end": "2021-11-20 08:44:36.500321", "rc": 0, "start": "2021-11-20 08:44:36.277270", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.889964", "end": "2021-11-20 08:44:36.500331", "rc": 0, "start": "2021-11-20 08:44:35.610367", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:44:36.546 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:36 +0000 (0:00:00.449) 0:02:10.492 ***** 2021-11-20T08:44:36.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:36 +0000 (0:00:00.031) 0:02:10.523 ***** 2021-11-20T08:44:40.933 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-11-20T08:44:40.935 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:40.956 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:40 +0000 (0:00:04.377) 0:02:14.901 ***** 2021-11-20T08:44:41.006 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:41.008 INFO:teuthology.task.ansible.out: 2021-11-20T08:44:41.029 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.073) 0:02:14.975 ***** 2021-11-20T08:44:41.036 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-20T08:44:41.036 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.060 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.030) 0:02:15.006 ***** 2021-11-20T08:44:41.090 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.115 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.055) 0:02:15.061 ***** 2021-11-20T08:44:41.134 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-11-20T08:44:41.135 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.160 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.044) 0:02:15.105 ***** 2021-11-20T08:44:41.180 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.204 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.042) 0:02:15.148 ***** 2021-11-20T08:44:41.231 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-11-20T08:44:41.232 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.254 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.051) 0:02:15.199 ***** 2021-11-20T08:44:41.271 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.295 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.041) 0:02:15.240 ***** 2021-11-20T08:44:41.337 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-20T08:44:41.338 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.360 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.064) 0:02:15.305 ***** 2021-11-20T08:44:41.388 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:41.414 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.053) 0:02:15.358 ***** 2021-11-20T08:44:41.915 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-11-20T08:44:41.916 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T08:44:41.937 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.524) 0:02:15.883 ***** 2021-11-20T08:44:41.945 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T08:44:41.971 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:41 +0000 (0:00:00.033) 0:02:15.916 ***** 2021-11-20T08:44:43.808 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-11-20T08:44:43.810 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:43.810 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:43.842 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:43 +0000 (0:00:01.870) 0:02:17.787 ***** 2021-11-20T08:44:43.876 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:43 +0000 (0:00:00.034) 0:02:17.821 ***** 2021-11-20T08:44:44.378 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-20T08:44:44.380 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T08:44:44.401 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T08:44:44.846 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T08:44:44.848 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T08:44:45.271 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T08:44:45.285 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T08:44:45.320 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:45 +0000 (0:00:01.443) 0:02:19.265 ***** 2021-11-20T08:44:45.347 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:45 +0000 (0:00:00.026) 0:02:19.292 ***** 2021-11-20T08:44:45.527 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:44:45.550 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:45 +0000 (0:00:00.203) 0:02:19.496 ***** 2021-11-20T08:44:45.558 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:44:45.583 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:45 +0000 (0:00:00.032) 0:02:19.529 ***** 2021-11-20T08:44:46.077 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T08:44:46.078 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T08:44:46.112 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T08:44:46.548 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T08:44:46.582 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T08:44:47.127 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397886.6259663-26659-20607165785399/source", "state": "file", "uid": 0} 2021-11-20T08:44:47.188 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:01.605) 0:02:21.134 ***** 2021-11-20T08:44:47.196 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397886.592855-26655-98407032138422/source", "state": "file", "uid": 0} 2021-11-20T08:44:47.229 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:00.040) 0:02:21.174 ***** 2021-11-20T08:44:47.236 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-20T08:44:47.238 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T08:44:47.261 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:00.031) 0:02:21.206 ***** 2021-11-20T08:44:47.268 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T08:44:47.294 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:00.030) 0:02:21.237 ***** 2021-11-20T08:44:47.453 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-20T08:44:47.455 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T08:44:47.477 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:00.185) 0:02:21.423 ***** 2021-11-20T08:44:47.497 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T08:44:47.524 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:00.046) 0:02:21.470 ***** 2021-11-20T08:44:47.532 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-20T08:44:47.534 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:47.535 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:47.546 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:47.571 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:00.046) 0:02:21.516 ***** 2021-11-20T08:44:47.579 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:47.591 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:47.592 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:47.602 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:47.626 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:47 +0000 (0:00:00.055) 0:02:21.572 ***** 2021-11-20T08:44:49.398 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-20T08:44:49.399 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:49.421 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:49 +0000 (0:00:01.794) 0:02:23.367 ***** 2021-11-20T08:44:49.459 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:49.482 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:49 +0000 (0:00:00.060) 0:02:23.427 ***** 2021-11-20T08:44:49.489 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-20T08:44:49.491 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:49.518 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:49 +0000 (0:00:00.036) 0:02:23.464 ***** 2021-11-20T08:44:49.524 INFO:teuthology.task.ansible.out: 2021-11-20T08:44:49.555 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:49.577 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:49 +0000 (0:00:00.058) 0:02:23.523 ***** 2021-11-20T08:44:49.971 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T08:44:49.994 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:49 +0000 (0:00:00.416) 0:02:23.939 ***** 2021-11-20T08:44:50.064 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T08:44:50.093 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:50 +0000 (0:00:00.098) 0:02:24.038 ***** 2021-11-20T08:44:50.466 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T08:44:50.467 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T08:44:50.496 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:50 +0000 (0:00:00.402) 0:02:24.441 ***** 2021-11-20T08:44:50.534 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-11-20T08:44:50.535 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:50.557 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:50 +0000 (0:00:00.061) 0:02:24.502 ***** 2021-11-20T08:44:50.574 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T08:44:50.576 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T08:44:50.598 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:50 +0000 (0:00:00.040) 0:02:24.543 ***** 2021-11-20T08:44:50.636 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi093.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:50.647 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T08:44:50.671 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:50 +0000 (0:00:00.073) 0:02:24.617 ***** 2021-11-20T08:44:50.752 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-11-20T08:44:50.754 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:44:50.780 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:50 +0000 (0:00:00.108) 0:02:24.725 ***** 2021-11-20T08:44:50.871 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:44:50.895 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:50 +0000 (0:00:00.114) 0:02:24.840 ***** 2021-11-20T08:44:51.015 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-20T08:44:51.017 INFO:teuthology.task.ansible.out:fatal: [smithi102.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T08:44:51.039 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:51 +0000 (0:00:00.144) 0:02:24.984 ***** 2021-11-20T08:44:51.077 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-20T08:44:51.079 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:51.101 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:51 +0000 (0:00:00.060) 0:02:25.045 ***** 2021-11-20T08:44:51.137 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-20T08:44:51.139 INFO:teuthology.task.ansible.out:fatal: [smithi093.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T08:44:51.162 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:51 +0000 (0:00:00.062) 0:02:25.107 ***** 2021-11-20T08:44:51.189 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:44:51.210 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:51 +0000 (0:00:00.048) 0:02:25.156 ***** 2021-11-20T08:44:51.563 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-11-20T08:44:51.565 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T08:44:51.589 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:51 +0000 (0:00:00.378) 0:02:25.534 ***** 2021-11-20T08:44:51.683 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T08:44:51.711 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:51 +0000 (0:00:00.122) 0:02:25.656 ***** 2021-11-20T08:44:52.266 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-20T08:44:52.269 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 08:41:22 UTC", "ActiveEnterTimestampMonotonic": "19444741", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target network-online.target -.mount system.slice tmp.mount var-run.mount basic.target systemd-tmpfiles-setup.service systemd-journald.socket sysinit.target time-sync.target nss-lookup.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 08:41:22 UTC", "AssertTimestampMonotonic": "19430026", "Before": "multi-user.target getty@tty1.service xdm.service plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 08:41:22 UTC", "Co 2021-11-20T08:44:52.271 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "19430025", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1264", "ExecMainStartTimestamp": "Sat 2021-11-20 08:41:22 UTC", "ExecMainStartTimestampMonotonic": "19444698", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 08:41:22 UTC] ; stop_time=[n/a] ; pid=1264 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 08:41:22 UTC", "InactiveExitTimestampMonotonic": "194447 2021-11-20T08:44:52.272 INFO:teuthology.task.ansible.out:41", "InvocationID": "512339443e664d7db251a1c6b684222c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1264", "MemoryAccounting": "yes", "MemoryCurrent": "679936", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T08:44:52.275 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 08:41:22 UTC", "StateChangeTimestampMonotonic": "19444741", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 08:41:22 UTC", "WatchdogTimestampMonotonic": "19444740", "WatchdogUSec": "0"}} 2021-11-20T08:44:52.289 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:52 +0000 (0:00:00.577) 0:02:26.234 ***** 2021-11-20T08:44:52.298 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 08:41:24 UTC", "ActiveEnterTimestampMonotonic": "20331081", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target network-online.target basic.target local-fs.target systemd-tmpfiles-setup.service time-sync.target sysinit.target -.mount system.slice var-run.mount tmp.mount systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 08:41:24 UTC", "AssertTimestampMonotonic": "20300412", "Before": "getty@tty1.service plymouth-quit.service xdm.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 08:41:24 UTC", "Co 2021-11-20T08:44:52.300 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "20300412", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1264", "ExecMainStartTimestamp": "Sat 2021-11-20 08:41:24 UTC", "ExecMainStartTimestampMonotonic": "20331053", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 08:41:24 UTC] ; stop_time=[n/a] ; pid=1264 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 08:41:24 UTC", "InactiveExitTimestampMonotonic": "203310 2021-11-20T08:44:52.302 INFO:teuthology.task.ansible.out:81", "InvocationID": "70c55f536004452d960d0c87e48a93b5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127241", "LimitNPROCSoft": "127241", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127241", "LimitSIGPENDINGSoft": "127241", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1264", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T08:44:52.303 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 08:41:24 UTC", "StateChangeTimestampMonotonic": "20331081", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203586", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 08:41:24 UTC", "WatchdogTimestampMonotonic": "20331080", "WatchdogUSec": "0"}} 2021-11-20T08:44:52.324 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:52 +0000 (0:00:00.035) 0:02:26.269 ***** 2021-11-20T08:44:52.477 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-11-20T08:44:52.479 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001788", "end": "2021-11-20 08:44:52.462131", "rc": 0, "start": "2021-11-20 08:44:52.460343", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T08:44:52.501 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:52 +0000 (0:00:00.177) 0:02:26.447 ***** 2021-11-20T08:44:52.519 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001767", "end": "2021-11-20 08:44:52.502744", "rc": 0, "start": "2021-11-20 08:44:52.500977", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T08:44:52.542 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:52 +0000 (0:00:00.040) 0:02:26.488 ***** 2021-11-20T08:44:54.324 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-11-20T08:44:54.326 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:54.347 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:54 +0000 (0:00:01.804) 0:02:28.292 ***** 2021-11-20T08:44:54.355 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:44:54.379 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:44:54 +0000 (0:00:00.032) 0:02:28.324 ***** 2021-11-20T08:45:34.051 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-11-20T08:45:34.053 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T08:45:34.073 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:34 +0000 (0:00:39.694) 0:03:08.019 ***** 2021-11-20T08:45:34.165 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T08:45:34.189 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:34 +0000 (0:00:00.115) 0:03:08.134 ***** 2021-11-20T08:45:34.755 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T08:45:34.777 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:34 +0000 (0:00:00.588) 0:03:08.723 ***** 2021-11-20T08:45:34.785 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T08:45:34.809 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:34 +0000 (0:00:00.031) 0:03:08.755 ***** 2021-11-20T08:45:50.513 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.508493", "end": "2021-11-20 08:45:50.493019", "failed_when_result": false, "rc": 0, "start": "2021-11-20 08:45:34.984526", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T08:45:50.535 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:50 +0000 (0:00:15.725) 0:03:24.480 ***** 2021-11-20T08:45:50.613 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.639810", "end": "2021-11-20 08:45:50.589770", "failed_when_result": false, "rc": 0, "start": "2021-11-20 08:45:34.949960", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T08:45:50.638 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:50 +0000 (0:00:00.102) 0:03:24.583 ***** 2021-11-20T08:45:51.012 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T08:45:51.033 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:51 +0000 (0:00:00.395) 0:03:24.979 ***** 2021-11-20T08:45:51.097 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T08:45:51.121 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:51 +0000 (0:00:00.087) 0:03:25.066 ***** 2021-11-20T08:45:51.233 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-20T08:45:51.235 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013529", "end": "2021-11-20 08:45:51.217000", "rc": 0, "start": "2021-11-20 08:45:51.203471", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:45:51.258 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:51 +0000 (0:00:00.136) 0:03:25.203 ***** 2021-11-20T08:45:51.327 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013517", "end": "2021-11-20 08:45:51.311596", "rc": 0, "start": "2021-11-20 08:45:51.298079", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:45:51.350 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:51 +0000 (0:00:00.093) 0:03:25.296 ***** 2021-11-20T08:45:51.464 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-11-20T08:45:51.466 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.007933", "end": "2021-11-20 08:45:51.444196", "rc": 0, "start": "2021-11-20 08:45:51.436263", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:45:51.489 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:51 +0000 (0:00:00.138) 0:03:25.434 ***** 2021-11-20T08:45:51.556 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.005168", "end": "2021-11-20 08:45:51.530646", "rc": 0, "start": "2021-11-20 08:45:51.525478", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:45:51.580 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:45:51 +0000 (0:00:00.090) 0:03:25.525 ***** 2021-11-20T08:46:07.202 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-11-20T08:46:07.204 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.513341", "end": "2021-11-20 08:46:07.174680", "rc": 0, "start": "2021-11-20 08:45:51.661339", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:46:07.226 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:15.646) 0:03:41.171 ***** 2021-11-20T08:46:07.265 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.481428", "end": "2021-11-20 08:46:07.241477", "rc": 0, "start": "2021-11-20 08:45:51.760049", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:46:07.289 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.062) 0:03:41.234 ***** 2021-11-20T08:46:07.432 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-20T08:46:07.434 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T08:46:07.456 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.167) 0:03:41.401 ***** 2021-11-20T08:46:07.495 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-11-20T08:46:07.497 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T08:46:07.518 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.061) 0:03:41.463 ***** 2021-11-20T08:46:07.525 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-20T08:46:07.526 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T08:46:07.561 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.042) 0:03:41.506 ***** 2021-11-20T08:46:07.572 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-20T08:46:07.574 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T08:46:07.599 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.037) 0:03:41.544 ***** 2021-11-20T08:46:07.606 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-11-20T08:46:07.608 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T08:46:07.632 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.032) 0:03:41.577 ***** 2021-11-20T08:46:07.650 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-20T08:46:07.652 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T08:46:07.653 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T08:46:07.675 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.043) 0:03:41.621 ***** 2021-11-20T08:46:07.683 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-20T08:46:07.685 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T08:46:07.710 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.034) 0:03:41.655 ***** 2021-11-20T08:46:07.718 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T08:46:07.719 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T08:46:07.721 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T08:46:07.745 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.034) 0:03:41.690 ***** 2021-11-20T08:46:07.763 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-20T08:46:07.766 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T08:46:07.767 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T08:46:07.792 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.047) 0:03:41.738 ***** 2021-11-20T08:46:07.841 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T08:46:07.843 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T08:46:07.845 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T08:46:07.865 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.073) 0:03:41.811 ***** 2021-11-20T08:46:07.934 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-20T08:46:07.936 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T08:46:07.968 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:07 +0000 (0:00:00.102) 0:03:41.913 ***** 2021-11-20T08:46:08.050 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T08:46:08.075 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.107) 0:03:42.021 ***** 2021-11-20T08:46:08.157 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-11-20T08:46:08.159 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T08:46:08.181 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.105) 0:03:42.126 ***** 2021-11-20T08:46:08.207 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-20T08:46:08.209 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:08.230 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.049) 0:03:42.175 ***** 2021-11-20T08:46:08.259 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-20T08:46:08.261 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:08.285 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.054) 0:03:42.230 ***** 2021-11-20T08:46:08.293 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-11-20T08:46:08.295 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T08:46:08.319 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.033) 0:03:42.264 ***** 2021-11-20T08:46:08.348 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-20T08:46:08.350 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:08.371 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.052) 0:03:42.316 ***** 2021-11-20T08:46:08.399 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-20T08:46:08.401 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:08.422 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.051) 0:03:42.367 ***** 2021-11-20T08:46:08.502 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-20T08:46:08.504 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-20T08:46:08.527 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.105) 0:03:42.472 ***** 2021-11-20T08:46:08.640 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-20T08:46:08.667 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:08 +0000 (0:00:00.139) 0:03:42.611 ***** 2021-11-20T08:46:09.075 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-20T08:46:09.077 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-20T08:46:09.102 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:09 +0000 (0:00:00.436) 0:03:43.048 ***** 2021-11-20T08:46:09.430 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi102"}, "changed": false} Saturday 20 November 2021 08:46:09 +0000 (0:00:00.063) 0:03:43.111 ***** TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} Saturday 20 November 2021 08:46:09 +0000 (0:00:00.097) 0:03:43.209 ***** TASK [testnode : Set hostname var] ********************************************* ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi093"}, "changed": false} Saturday 20 November 2021 08:46:09 +0000 (0:00:00.090) 0:03:43.300 ***** 2021-11-20T08:46:09.972 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-11-20T08:46:09.974 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi093.front.sepia.ceph.com", "ansible_hostname": "smithi093", "ansible_nodename": "smithi093"}, "changed": true, "name": "smithi093"} 2021-11-20T08:46:09.996 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:09 +0000 (0:00:00.640) 0:03:43.940 ***** 2021-11-20T08:46:10.215 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi102.front.sepia.ceph.com", "ansible_hostname": "smithi102", "ansible_nodename": "smithi102"}, "changed": true, "name": "smithi102"} 2021-11-20T08:46:10.239 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:10 +0000 (0:00:00.243) 0:03:44.184 ***** 2021-11-20T08:46:10.456 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-11-20T08:46:10.457 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T08:46:10.479 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:10 +0000 (0:00:00.240) 0:03:44.424 ***** 2021-11-20T08:46:10.697 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T08:46:10.721 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:10 +0000 (0:00:00.242) 0:03:44.667 ***** 2021-11-20T08:46:10.950 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-11-20T08:46:10.952 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T08:46:10.975 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:10 +0000 (0:00:00.252) 0:03:44.919 ***** 2021-11-20T08:46:11.173 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T08:46:11.196 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:11 +0000 (0:00:00.222) 0:03:45.141 ***** 2021-11-20T08:46:11.497 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-11-20T08:46:11.499 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T08:46:11.522 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:11 +0000 (0:00:00.325) 0:03:45.467 ***** 2021-11-20T08:46:11.720 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T08:46:11.743 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:11 +0000 (0:00:00.221) 0:03:45.689 ***** 2021-11-20T08:46:11.993 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-20T08:46:11.995 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T08:46:12.016 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.272) 0:03:45.962 ***** 2021-11-20T08:46:12.213 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T08:46:12.215 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T08:46:12.238 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.220) 0:03:46.182 ***** 2021-11-20T08:46:12.244 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-20T08:46:12.245 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T08:46:12.269 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.033) 0:03:46.215 ***** 2021-11-20T08:46:12.434 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T08:46:12.436 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T08:46:12.460 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.189) 0:03:46.404 ***** 2021-11-20T08:46:12.477 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T08:46:12.480 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T08:46:12.503 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.043) 0:03:46.448 ***** 2021-11-20T08:46:12.677 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T08:46:12.678 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T08:46:12.701 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.198) 0:03:46.646 ***** 2021-11-20T08:46:12.709 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T08:46:12.734 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.032) 0:03:46.678 ***** 2021-11-20T08:46:12.950 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T08:46:12.951 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T08:46:12.973 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:12 +0000 (0:00:00.239) 0:03:46.918 ***** 2021-11-20T08:46:13.184 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T08:46:13.208 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:13 +0000 (0:00:00.235) 0:03:47.153 ***** 2021-11-20T08:46:13.448 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T08:46:13.472 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:13 +0000 (0:00:00.263) 0:03:47.417 ***** 2021-11-20T08:46:13.692 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T08:46:13.693 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T08:46:13.715 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:13 +0000 (0:00:00.242) 0:03:47.660 ***** 2021-11-20T08:46:13.948 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T08:46:13.970 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:13 +0000 (0:00:00.255) 0:03:47.916 ***** 2021-11-20T08:46:16.287 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T08:46:16.289 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.388442", "end": "2021-11-20 08:46:16.269438", "rc": 0, "start": "2021-11-20 08:46:13.880996", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:46:16.311 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:16 +0000 (0:00:02.340) 0:03:50.256 ***** 2021-11-20T08:46:16.516 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-20T08:46:16.518 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002579", "end": "2021-11-20 08:46:16.497860", "rc": 0, "start": "2021-11-20 08:46:16.495281", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:46:16.539 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:16 +0000 (0:00:00.228) 0:03:50.484 ***** 2021-11-20T08:46:16.587 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T08:46:16.610 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:16 +0000 (0:00:00.070) 0:03:50.555 ***** 2021-11-20T08:46:16.658 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-20T08:46:16.660 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T08:46:16.681 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:16 +0000 (0:00:00.071) 0:03:50.627 ***** 2021-11-20T08:46:16.709 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T08:46:16.711 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.552364", "end": "2021-11-20 08:46:16.690704", "rc": 0, "start": "2021-11-20 08:46:14.138340", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:46:16.734 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:16 +0000 (0:00:00.052) 0:03:50.679 ***** 2021-11-20T08:46:16.949 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-20T08:46:16.951 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002590", "end": "2021-11-20 08:46:16.925692", "rc": 0, "start": "2021-11-20 08:46:16.923102", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:46:16.975 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:16 +0000 (0:00:00.240) 0:03:50.920 ***** 2021-11-20T08:46:17.012 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:46:17.014 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T08:46:17.034 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:17 +0000 (0:00:00.059) 0:03:50.979 ***** 2021-11-20T08:46:17.081 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-20T08:46:17.083 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T08:46:17.105 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:17 +0000 (0:00:00.070) 0:03:51.050 ***** 2021-11-20T08:46:17.164 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** ok: [smithi093.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T08:46:17.595 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T08:46:17.606 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T08:46:18.027 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T08:46:18.039 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T08:46:18.470 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T08:46:18.513 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397978.0655339-2126-209227289605537/source", "state": "file", "uid": 0} 2021-11-20T08:46:18.537 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:18 +0000 (0:00:01.432) 0:03:52.483 ***** 2021-11-20T08:46:18.951 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397978.4957042-2174-268089142748147/source", "state": "file", "uid": 0} 2021-11-20T08:46:18.977 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:18 +0000 (0:00:00.439) 0:03:52.923 ***** 2021-11-20T08:46:19.029 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi093.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T08:46:19.463 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T08:46:19.517 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397979.0555062-2345-164629932896754/source", "state": "file", "uid": 0} 2021-11-20T08:46:19.543 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:19 +0000 (0:00:00.565) 0:03:53.488 ***** 2021-11-20T08:46:19.953 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397979.4939742-2390-213009251458281/source", "state": "file", "uid": 0} 2021-11-20T08:46:19.979 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:19 +0000 (0:00:00.434) 0:03:53.923 ***** 2021-11-20T08:46:20.017 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-20T08:46:20.019 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T08:46:20.458 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T08:46:20.459 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T08:46:20.898 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T08:46:20.910 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T08:46:21.339 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T08:46:21.403 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397980.9500659-2451-74120353082774/source", "state": "file", "uid": 0} 2021-11-20T08:46:21.438 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:21 +0000 (0:00:01.460) 0:03:55.384 ***** 2021-11-20T08:46:21.835 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637397981.36992-2520-227652399736383/source", "state": "file", "uid": 0} 2021-11-20T08:46:21.863 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:21 +0000 (0:00:00.424) 0:03:55.808 ***** 2021-11-20T08:46:21.923 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-11-20T08:46:21.924 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.272612", "end": "2021-11-20 08:46:21.896297", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 08:46:21.623685", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-11-20T08:46:21.925 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-11-20T08:46:21.950 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:21 +0000 (0:00:00.086) 0:03:55.895 ***** 2021-11-20T08:46:22.332 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.262439", "end": "2021-11-20 08:46:22.306411", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 08:46:22.043972", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-11-20T08:46:22.356 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:22 +0000 (0:00:00.406) 0:03:56.302 ***** 2021-11-20T08:46:29.304 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-11-20T08:46:29.306 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:46:29.326 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:29 +0000 (0:00:06.969) 0:04:03.271 ***** 2021-11-20T08:46:29.375 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:46:29.399 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:29 +0000 (0:00:00.073) 0:04:03.345 ***** 2021-11-20T08:46:30.100 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-11-20T08:46:30.102 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T08:46:30.122 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T08:46:30.575 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T08:46:30.586 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T08:46:31.813 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T08:46:31.836 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:31 +0000 (0:00:02.437) 0:04:05.782 ***** 2021-11-20T08:46:32.021 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T08:46:32.046 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.209) 0:04:05.991 ***** 2021-11-20T08:46:32.221 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-20T08:46:32.223 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.183503", "end": "2021-11-20 08:46:32.201751", "rc": 0, "start": "2021-11-20 08:46:32.018248", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T08:46:32.245 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.199) 0:04:06.191 ***** 2021-11-20T08:46:32.283 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:46:32.284 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T08:46:32.305 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.059) 0:04:06.250 ***** 2021-11-20T08:46:32.342 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-20T08:46:32.343 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:32.364 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.059) 0:04:06.309 ***** 2021-11-20T08:46:32.401 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:32.423 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.059) 0:04:06.369 ***** 2021-11-20T08:46:32.430 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-20T08:46:32.432 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.180820", "end": "2021-11-20 08:46:32.412192", "rc": 0, "start": "2021-11-20 08:46:32.231372", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T08:46:32.456 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.031) 0:04:06.400 ***** 2021-11-20T08:46:32.492 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:46:32.494 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T08:46:32.514 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.059) 0:04:06.460 ***** 2021-11-20T08:46:32.542 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-20T08:46:32.544 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:32.563 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.049) 0:04:06.509 ***** 2021-11-20T08:46:32.590 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-11-20T08:46:32.592 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:32.612 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.047) 0:04:06.557 ***** 2021-11-20T08:46:32.628 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T08:46:32.630 INFO:teuthology.task.ansible.out:fatal: [smithi102.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007308", "end": "2021-11-20 08:46:32.600831", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 08:46:32.593523", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T08:46:32.652 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.040) 0:04:06.598 ***** 2021-11-20T08:46:32.701 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-20T08:46:32.703 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:32.723 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.070) 0:04:06.668 ***** 2021-11-20T08:46:32.813 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T08:46:32.815 INFO:teuthology.task.ansible.out:fatal: [smithi093.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007528", "end": "2021-11-20 08:46:32.795748", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 08:46:32.788220", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T08:46:32.842 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.119) 0:04:06.788 ***** 2021-11-20T08:46:32.870 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-20T08:46:32.872 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:46:32.903 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:32 +0000 (0:00:00.058) 0:04:06.846 ***** 2021-11-20T08:46:50.836 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T08:46:50.838 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T08:46:50.859 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:50 +0000 (0:00:17.958) 0:04:24.804 ***** 2021-11-20T08:46:50.981 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T08:46:51.008 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:51 +0000 (0:00:00.149) 0:04:24.953 ***** 2021-11-20T08:46:53.851 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T08:46:53.853 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T08:46:53.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:53 +0000 (0:00:02.866) 0:04:27.820 ***** 2021-11-20T08:46:54.063 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T08:46:54.086 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:46:54 +0000 (0:00:00.211) 0:04:28.032 ***** 2021-11-20T08:47:03.082 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T08:47:03.084 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T08:47:03.107 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:03 +0000 (0:00:09.020) 0:04:37.052 ***** 2021-11-20T08:47:03.115 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T08:47:03.144 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:03 +0000 (0:00:00.037) 0:04:37.090 ***** 2021-11-20T08:47:03.152 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-11-20T08:47:03.154 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:03.181 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:03 +0000 (0:00:00.035) 0:04:37.126 ***** 2021-11-20T08:47:03.188 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:03.215 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:03 +0000 (0:00:00.034) 0:04:37.160 ***** 2021-11-20T08:47:05.301 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T08:47:05.304 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:47:05.326 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:05 +0000 (0:00:02.111) 0:04:39.271 ***** 2021-11-20T08:47:05.354 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:47:05.378 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:05 +0000 (0:00:00.051) 0:04:39.323 ***** 2021-11-20T08:47:13.047 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T08:47:13.049 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T08:47:13.071 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:13 +0000 (0:00:07.692) 0:04:47.016 ***** 2021-11-20T08:47:13.362 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T08:47:13.385 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:13 +0000 (0:00:00.314) 0:04:47.330 ***** 2021-11-20T08:47:13.392 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-20T08:47:13.393 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target basic.target system.slice dbus.service dbus.socket polkit.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service ebtables.service iptables.service shutdown.target ipset.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T08:47:13.395 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T08:47:13.397 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T08:47:13.398 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T08:47:13.418 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:13 +0000 (0:00:00.032) 0:04:47.363 ***** 2021-11-20T08:47:13.687 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket polkit.service dbus.service sysinit.target system.slice basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "nftables.service iptables.service shutdown.target ip6tables.service ipset.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T08:47:13.688 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T08:47:13.690 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127241", "LimitNPROCSoft": "127241", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127241", "LimitSIGPENDINGSoft": "127241", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target dbus.socket system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T08:47:13.692 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203586", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T08:47:13.711 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:13 +0000 (0:00:00.292) 0:04:47.656 ***** 2021-11-20T08:47:13.719 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi102.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-20T08:47:13.744 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:13 +0000 (0:00:00.033) 0:04:47.690 ***** 2021-11-20T08:47:14.002 INFO:teuthology.task.ansible.out:fatal: [smithi093.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-20T08:47:14.024 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.279) 0:04:47.969 ***** 2021-11-20T08:47:14.397 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T08:47:14.397 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T08:47:14.419 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.395) 0:04:48.364 ***** 2021-11-20T08:47:14.440 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.020) 0:04:48.385 ***** 2021-11-20T08:47:14.458 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.485 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.039) 0:04:48.425 ***** 2021-11-20T08:47:14.487 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.511 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.030) 0:04:48.456 ***** 2021-11-20T08:47:14.517 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-20T08:47:14.519 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.543 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.031) 0:04:48.487 ***** 2021-11-20T08:47:14.549 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.574 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.032) 0:04:48.519 ***** 2021-11-20T08:47:14.581 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.606 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.030) 0:04:48.550 ***** 2021-11-20T08:47:14.611 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.637 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.030) 0:04:48.581 ***** 2021-11-20T08:47:14.641 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-20T08:47:14.643 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.666 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.030) 0:04:48.611 ***** 2021-11-20T08:47:14.668 INFO:teuthology.task.ansible.out: 2021-11-20T08:47:14.675 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.700 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.033) 0:04:48.645 ***** 2021-11-20T08:47:14.938 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 08:47:14 +0000 (0:00:00.030) 0:04:48.676 ***** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 08:47:14 +0000 (0:00:00.031) 0:04:48.708 ***** TASK [testnode : Check for custom repos] *************************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 08:47:14 +0000 (0:00:00.030) 0:04:48.738 ***** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 08:47:14 +0000 (0:00:00.029) 0:04:48.767 ***** TASK [testnode : Remove custom repos] ****************************************** Saturday 20 November 2021 08:47:14 +0000 (0:00:00.036) 0:04:48.804 ***** Saturday 20 November 2021 08:47:14 +0000 (0:00:00.034) 0:04:48.838 ***** TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 08:47:14 +0000 (0:00:00.032) 0:04:48.871 ***** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.958 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.032) 0:04:48.903 ***** 2021-11-20T08:47:14.965 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-20T08:47:14.967 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:14.992 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:14 +0000 (0:00:00.034) 0:04:48.938 ***** 2021-11-20T08:47:15.001 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.032 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.040) 0:04:48.978 ***** 2021-11-20T08:47:15.039 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-11-20T08:47:15.041 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.051 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.076 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.043) 0:04:49.021 ***** 2021-11-20T08:47:15.085 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.111 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.035) 0:04:49.056 ***** 2021-11-20T08:47:15.118 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-20T08:47:15.119 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.144 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.087 ***** 2021-11-20T08:47:15.149 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.175 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.032) 0:04:49.119 ***** 2021-11-20T08:47:15.181 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-20T08:47:15.203 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.029) 0:04:49.149 ***** 2021-11-20T08:47:15.233 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.029) 0:04:49.179 ***** 2021-11-20T08:47:15.239 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-11-20T08:47:15.241 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.262 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.028) 0:04:49.207 ***** 2021-11-20T08:47:15.268 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.293 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.239 ***** 2021-11-20T08:47:15.300 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-11-20T08:47:15.302 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.324 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.269 ***** 2021-11-20T08:47:15.331 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.355 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.301 ***** 2021-11-20T08:47:15.362 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-11-20T08:47:15.364 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.386 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.332 ***** 2021-11-20T08:47:15.394 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.418 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.364 ***** 2021-11-20T08:47:15.426 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.450 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.395 ***** 2021-11-20T08:47:15.457 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.480 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.426 ***** 2021-11-20T08:47:15.487 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-20T08:47:15.489 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.510 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.029) 0:04:49.456 ***** 2021-11-20T08:47:15.517 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.544 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.033) 0:04:49.489 ***** 2021-11-20T08:47:15.550 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T08:47:15.552 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.575 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.520 ***** 2021-11-20T08:47:15.581 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.605 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.550 ***** 2021-11-20T08:47:15.611 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-20T08:47:15.613 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.634 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.029) 0:04:49.580 ***** 2021-11-20T08:47:15.642 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.665 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.611 ***** 2021-11-20T08:47:15.672 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-20T08:47:15.674 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.694 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.028) 0:04:49.640 ***** 2021-11-20T08:47:15.700 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.723 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.028) 0:04:49.668 ***** 2021-11-20T08:47:15.729 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-20T08:47:15.730 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.753 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.699 ***** 2021-11-20T08:47:15.759 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.785 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.730 ***** 2021-11-20T08:47:15.791 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-11-20T08:47:15.793 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.815 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.760 ***** 2021-11-20T08:47:15.832 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.856 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.040) 0:04:49.801 ***** 2021-11-20T08:47:15.863 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-20T08:47:15.865 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.887 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.832 ***** 2021-11-20T08:47:15.896 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.919 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.031) 0:04:49.864 ***** 2021-11-20T08:47:15.926 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-20T08:47:15.928 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.949 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.895 ***** 2021-11-20T08:47:15.956 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:15.980 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:15 +0000 (0:00:00.030) 0:04:49.925 ***** 2021-11-20T08:47:15.987 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-11-20T08:47:15.989 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.010 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:49.956 ***** 2021-11-20T08:47:16.018 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.044 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.032) 0:04:49.988 ***** 2021-11-20T08:47:16.050 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-11-20T08:47:16.051 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.073 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.018 ***** 2021-11-20T08:47:16.080 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.103 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.048 ***** 2021-11-20T08:47:16.110 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-20T08:47:16.112 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.133 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.079 ***** 2021-11-20T08:47:16.140 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.164 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.109 ***** 2021-11-20T08:47:16.170 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T08:47:16.172 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.194 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.139 ***** 2021-11-20T08:47:16.201 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.225 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.170 ***** 2021-11-20T08:47:16.232 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T08:47:16.234 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.256 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.201 ***** 2021-11-20T08:47:16.262 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.291 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.034) 0:04:50.236 ***** 2021-11-20T08:47:16.298 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T08:47:16.301 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.323 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.032) 0:04:50.269 ***** 2021-11-20T08:47:16.331 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.356 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.032) 0:04:50.301 ***** 2021-11-20T08:47:16.363 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T08:47:16.365 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.389 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.033) 0:04:50.334 ***** 2021-11-20T08:47:16.396 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.420 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.365 ***** 2021-11-20T08:47:16.425 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T08:47:16.427 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.449 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.029) 0:04:50.395 ***** 2021-11-20T08:47:16.456 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.479 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.425 ***** 2021-11-20T08:47:16.486 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T08:47:16.487 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.511 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.031) 0:04:50.457 ***** 2021-11-20T08:47:16.518 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.542 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.487 ***** 2021-11-20T08:47:16.549 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T08:47:16.551 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.574 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.031) 0:04:50.519 ***** 2021-11-20T08:47:16.581 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.606 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.033) 0:04:50.552 ***** 2021-11-20T08:47:16.614 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T08:47:16.616 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.638 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.031) 0:04:50.583 ***** 2021-11-20T08:47:16.645 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.671 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.032) 0:04:50.616 ***** 2021-11-20T08:47:16.678 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:47:16.680 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.701 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.647 ***** 2021-11-20T08:47:16.709 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.734 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.032) 0:04:50.679 ***** 2021-11-20T08:47:16.741 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T08:47:16.743 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.766 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.032) 0:04:50.711 ***** 2021-11-20T08:47:16.775 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.803 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.037) 0:04:50.749 ***** 2021-11-20T08:47:16.814 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T08:47:16.817 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.846 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.043) 0:04:50.792 ***** 2021-11-20T08:47:16.855 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.879 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.032) 0:04:50.825 ***** 2021-11-20T08:47:16.888 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T08:47:16.890 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.913 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.033) 0:04:50.858 ***** 2021-11-20T08:47:16.921 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.949 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.035) 0:04:50.894 ***** 2021-11-20T08:47:16.955 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T08:47:16.957 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:16.979 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:16 +0000 (0:00:00.030) 0:04:50.924 ***** 2021-11-20T08:47:16.987 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.011 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.032) 0:04:50.957 ***** 2021-11-20T08:47:17.207 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T08:47:17.209 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-20T08:47:17.233 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.222) 0:04:51.179 ***** 2021-11-20T08:47:17.241 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-20T08:47:17.265 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.031) 0:04:51.210 ***** 2021-11-20T08:47:17.450 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:47:17.473 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.208) 0:04:51.418 ***** 2021-11-20T08:47:17.492 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:47:17.517 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.044) 0:04:51.462 ***** 2021-11-20T08:47:17.671 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T08:47:17.673 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T08:47:17.695 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.178) 0:04:51.641 ***** 2021-11-20T08:47:17.724 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T08:47:17.727 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.747 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.051) 0:04:51.693 ***** 2021-11-20T08:47:17.768 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.021) 0:04:51.714 ***** 2021-11-20T08:47:17.786 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T08:47:17.788 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.810 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.041) 0:04:51.755 ***** 2021-11-20T08:47:17.819 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T08:47:17.820 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.844 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.034) 0:04:51.790 ***** 2021-11-20T08:47:17.850 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T08:47:17.853 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.032) 0:04:51.822 ***** 2021-11-20T08:47:17.884 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T08:47:17.886 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.908 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.030) 0:04:51.853 ***** 2021-11-20T08:47:17.915 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T08:47:17.916 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.939 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.031) 0:04:51.884 ***** 2021-11-20T08:47:17.946 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T08:47:17.948 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:17.970 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.030) 0:04:51.915 ***** 2021-11-20T08:47:17.976 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T08:47:17.978 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.001 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:17 +0000 (0:00:00.031) 0:04:51.946 ***** 2021-11-20T08:47:18.008 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T08:47:18.009 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.031 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.030) 0:04:51.977 ***** 2021-11-20T08:47:18.039 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T08:47:18.039 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.033) 0:04:52.010 ***** 2021-11-20T08:47:18.073 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.097 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.030) 0:04:52.041 ***** 2021-11-20T08:47:18.103 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T08:47:18.105 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.128 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.032) 0:04:52.074 ***** 2021-11-20T08:47:18.148 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.172 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.043) 0:04:52.117 ***** 2021-11-20T08:47:18.180 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T08:47:18.182 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.206 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.034) 0:04:52.151 ***** 2021-11-20T08:47:18.213 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T08:47:18.216 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.238 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.030) 0:04:52.182 ***** 2021-11-20T08:47:18.244 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T08:47:18.246 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.274 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.034) 0:04:52.217 ***** 2021-11-20T08:47:18.278 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T08:47:18.280 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.306 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.034) 0:04:52.251 ***** 2021-11-20T08:47:18.312 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-20T08:47:18.314 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.338 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.032) 0:04:52.283 ***** 2021-11-20T08:47:18.345 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T08:47:18.347 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.369 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.031) 0:04:52.315 ***** 2021-11-20T08:47:18.376 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T08:47:18.378 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.402 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.031) 0:04:52.347 ***** 2021-11-20T08:47:18.410 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-20T08:47:18.411 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.033) 0:04:52.380 ***** 2021-11-20T08:47:18.443 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T08:47:18.445 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.474 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.039) 0:04:52.420 ***** 2021-11-20T08:47:18.483 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T08:47:18.486 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.510 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.035) 0:04:52.455 ***** 2021-11-20T08:47:18.517 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T08:47:18.519 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.542 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.031) 0:04:52.487 ***** 2021-11-20T08:47:18.549 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T08:47:18.550 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.572 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.030) 0:04:52.517 ***** 2021-11-20T08:47:18.578 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.605 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.032) 0:04:52.550 ***** 2021-11-20T08:47:18.612 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.643 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.038) 0:04:52.589 ***** 2021-11-20T08:47:18.650 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-11-20T08:47:18.652 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.678 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.034) 0:04:52.623 ***** 2021-11-20T08:47:18.687 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-11-20T08:47:18.689 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.710 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.031) 0:04:52.655 ***** 2021-11-20T08:47:18.716 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T08:47:18.717 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.740 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.030) 0:04:52.685 ***** 2021-11-20T08:47:18.747 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-11-20T08:47:18.749 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.773 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.030) 0:04:52.716 ***** 2021-11-20T08:47:18.777 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.802 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.031) 0:04:52.747 ***** 2021-11-20T08:47:18.808 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T08:47:18.810 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.838 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.036) 0:04:52.783 ***** 2021-11-20T08:47:18.845 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-11-20T08:47:18.846 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.870 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.031) 0:04:52.815 ***** 2021-11-20T08:47:18.876 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-11-20T08:47:18.878 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.907 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.037) 0:04:52.852 ***** 2021-11-20T08:47:18.915 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-20T08:47:18.917 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.950 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.043) 0:04:52.895 ***** 2021-11-20T08:47:18.959 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-11-20T08:47:18.961 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:18.986 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:18 +0000 (0:00:00.035) 0:04:52.931 ***** 2021-11-20T08:47:19.003 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.026 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.039) 0:04:52.971 ***** 2021-11-20T08:47:19.034 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-20T08:47:19.036 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.057 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.031) 0:04:53.003 ***** 2021-11-20T08:47:19.064 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T08:47:19.065 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.092 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.034) 0:04:53.037 ***** 2021-11-20T08:47:19.099 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-11-20T08:47:19.101 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.124 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.032) 0:04:53.069 ***** 2021-11-20T08:47:19.132 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-20T08:47:19.133 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.157 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.032) 0:04:53.102 ***** 2021-11-20T08:47:19.165 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T08:47:19.166 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.190 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.033) 0:04:53.136 ***** 2021-11-20T08:47:19.197 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:47:19.198 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-20T08:47:19.222 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.031) 0:04:53.167 ***** 2021-11-20T08:47:19.230 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.253 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.031) 0:04:53.198 ***** 2021-11-20T08:47:19.270 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-20T08:47:19.272 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.282 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-20T08:47:19.283 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} TASK [testnode : set_fact] ***************************************************** ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-20T08:47:19.305 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.052) 0:04:53.250 ***** 2021-11-20T08:47:19.310 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-20T08:47:19.313 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.342 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.037) 0:04:53.288 ***** 2021-11-20T08:47:19.351 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.362 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-20T08:47:19.364 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.374 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.376 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.405 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.061) 0:04:53.350 ***** 2021-11-20T08:47:19.563 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-20T08:47:19.565 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003788", "end": "2021-11-20 08:47:19.534871", "rc": 0, "start": "2021-11-20 08:47:19.531083", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T08:47:19.595 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.191) 0:04:53.541 ***** 2021-11-20T08:47:19.627 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-11-20T08:47:19.629 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.656 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.060) 0:04:53.601 ***** 2021-11-20T08:47:19.666 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-20T08:47:19.667 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.005775", "end": "2021-11-20 08:47:19.603776", "rc": 0, "start": "2021-11-20 08:47:19.598001", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T08:47:19.696 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.040) 0:04:53.642 ***** 2021-11-20T08:47:19.703 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-20T08:47:19.704 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.727 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.030) 0:04:53.672 ***** 2021-11-20T08:47:19.735 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.761 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.034) 0:04:53.707 ***** 2021-11-20T08:47:19.769 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-11-20T08:47:19.771 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.794 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.031) 0:04:53.739 ***** 2021-11-20T08:47:19.803 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.828 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.034) 0:04:53.773 ***** 2021-11-20T08:47:19.836 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.862 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.033) 0:04:53.807 ***** 2021-11-20T08:47:19.874 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-11-20T08:47:19.876 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.910 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.048) 0:04:53.856 ***** 2021-11-20T08:47:19.955 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:19.982 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:19 +0000 (0:00:00.069) 0:04:53.925 ***** 2021-11-20T08:47:21.900 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:47:21.922 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:21 +0000 (0:00:01.942) 0:04:55.868 ***** 2021-11-20T08:47:21.971 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T08:47:21.972 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140160000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085000, 'block_used': 13010108, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085000, "block_size": 4096, "block_total": 240095108, "block_used": 13010108, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140160000, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T08:47:21.994 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:21 +0000 (0:00:00.072) 0:04:55.940 ***** 2021-11-20T08:47:22.065 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-11-20T08:47:22.066 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:47:22.091 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.095) 0:04:56.036 ***** 2021-11-20T08:47:22.128 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T08:47:22.130 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140184576, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085006, 'block_used': 13010102, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085006, "block_size": 4096, "block_total": 240095108, "block_used": 13010102, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140184576, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T08:47:22.154 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.063) 0:04:56.099 ***** 2021-11-20T08:47:22.193 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-11-20T08:47:22.195 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005231", "end": "2021-11-20 08:47:22.167782", "rc": 0, "start": "2021-11-20 08:47:22.162551", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:22.217 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.062) 0:04:56.162 ***** 2021-11-20T08:47:22.268 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-11-20T08:47:22.270 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140160000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085000, 'block_used': 13010108, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085000, "block_size": 4096, "block_total": 240095108, "block_used": 13010108, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140160000, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:22.297 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.079) 0:04:56.242 ***** 2021-11-20T08:47:22.324 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T08:47:22.357 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.061) 0:04:56.303 ***** 2021-11-20T08:47:22.369 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005253", "end": "2021-11-20 08:47:22.330599", "rc": 0, "start": "2021-11-20 08:47:22.325346", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:22.402 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.042) 0:04:56.345 ***** 2021-11-20T08:47:22.445 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-11-20T08:47:22.447 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140184576, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085006, 'block_used': 13010102, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085006, "block_size": 4096, "block_total": 240095108, "block_used": 13010102, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140184576, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:22.478 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.077) 0:04:56.423 ***** 2021-11-20T08:47:22.505 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T08:47:22.533 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:22 +0000 (0:00:00.055) 0:04:56.479 ***** 2021-11-20T08:47:24.157 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T08:47:24.159 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYMy2Smc6KniqO9QcZBFSz0iD870sQcciE'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.556768", "end": "2021-11-20 08:47:24.130980", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYMy2Smc6KniqO9QcZBFSz0iD870sQcciE"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:22.574212", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:24.327 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJkMd4780TxMogesIHF6qK8SaQx7KT5XJx'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.567056", "end": "2021-11-20 08:47:24.285238", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJkMd4780TxMogesIHF6qK8SaQx7KT5XJx"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:22.718182", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:25.456 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-NQBaOY-ggvG-sSws-P3O2-7CzV-miF2-sz7411', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040150400BGN', 'nvme-nvme.8086-50484654363230343031353034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.099664", "end": "2021-11-20 08:47:25.419746", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-NQBaOY-ggvG-sSws-P3O2-7CzV-miF2-sz7411", "nvme-INTEL_SSDPEDMD400G4_PHFT62040150400BGN", "nvme-nvme.8086-50484654363230343031353034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:24.320082", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-11-20T08:47:25.458 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:25.561 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-mci9wg-ZWzb-PVEO-TcnL-8Oc1-TfUJ-LPKjT7', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN', 'nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.053840", "end": "2021-11-20 08:47:25.532923", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-mci9wg-ZWzb-PVEO-TcnL-8Oc1-TfUJ-LPKjT7", "nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN", "nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:24.479083", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-11-20T08:47:25.564 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:26.707 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYOpSZlXnUdalZPcyG9dio6mNzVuMlZF0W'], 'uuids': ['efc7b32b-bdbf-4035-a570-287e69ca28ca'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.062562", "end": "2021-11-20 08:47:26.679995", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYOpSZlXnUdalZPcyG9dio6mNzVuMlZF0W"], "labels": [], "masters": [], "uuids": ["efc7b32b-bdbf-4035-a570-287e69ca28ca"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:25.617433", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:26.804 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJz37z8y2TQJZGC44x2ND1Xm4WXCfO9N8W'], 'uuids': ['7db59f61-c64d-4278-8d4c-dc274028b4a8'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.046643", "end": "2021-11-20 08:47:26.775779", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJz37z8y2TQJZGC44x2ND1Xm4WXCfO9N8W"], "labels": [], "masters": [], "uuids": ["7db59f61-c64d-4278-8d4c-dc274028b4a8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:25.729136", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:27.967 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDY1fWsvvj8cyD3h2EN7agds9TmaNm2JyZg'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.065987", "end": "2021-11-20 08:47:27.929940", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDY1fWsvvj8cyD3h2EN7agds9TmaNm2JyZg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:26.863953", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:28.044 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJdbqZdphZ0Q2Vjre9wuc1Y0aa5OP88S9E'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.053480", "end": "2021-11-20 08:47:28.016908", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJdbqZdphZ0Q2Vjre9wuc1Y0aa5OP88S9E"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:26.963428", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:29.225 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYihimHZCHNwSLUciYspfTKfw4lXuBZ4NK'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.058374", "end": "2021-11-20 08:47:29.195452", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYihimHZCHNwSLUciYspfTKfw4lXuBZ4NK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:28.137078", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:29.237 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5Z8', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5Z8', 'scsi-35000c500918e738f', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8', 'wwn-0x5000c500918e738f'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5L5Z8', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918e738f', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5Z8-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5Z8-part1', 'scsi-35000c500918e738f-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1', 'wwn-0x5000c500918e738f-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5Z8", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5Z8", "scsi-35000c500918e738f", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8", "wwn-0x5000c500918e738f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5Z8-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T08:47:29.238 INFO:teuthology.task.ansible.out:Z1W5L5Z8-part1", "scsi-35000c500918e738f-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1", "wwn-0x5000c500918e738f-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L5Z8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e738f"}}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:29.322 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJHRScsnJdlK0ljA9tT3kDCem0d7T8OqYj'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.085463", "end": "2021-11-20 08:47:29.290828", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJHRScsnJdlK0ljA9tT3kDCem0d7T8OqYj"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:28.205365", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:29.323 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-35000c50091e0bcf8', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM', 'wwn-0x5000c50091e0bcf8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QPPM', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0bcf8', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-35000c50091e0bcf8-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'wwn-0x5000c50091e0bcf8-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-35000c50091e0bcf8", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM", "wwn-0x5000c50091e0bcf8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T08:47:29.328 INFO:teuthology.task.ansible.out:Z1W5QPPM-part1", "scsi-35000c50091e0bcf8-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "wwn-0x5000c50091e0bcf8-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QPPM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0bcf8"}}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:30.471 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYq1OrsVIJgECpoif1Xk5ZR4vOqa3dvYz8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.055999", "end": "2021-11-20 08:47:30.447167", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYq1OrsVIJgECpoif1Xk5ZR4vOqa3dvYz8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:29.391168", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:30.511 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:30 +0000 (0:00:07.977) 0:05:04.457 ***** 2021-11-20T08:47:30.594 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJ8GGziliEt4VCHHz8FDcf0wTKTwjmD4tf'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.086395", "end": "2021-11-20 08:47:30.567705", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJ8GGziliEt4VCHHz8FDcf0wTKTwjmD4tf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:29.481310", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T08:47:30.625 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:30 +0000 (0:00:00.113) 0:05:04.570 ***** 2021-11-20T08:47:30.872 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T08:47:30.873 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYMy2Smc6KniqO9QcZBFSz0iD870sQcciE'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.141520", "end": "2021-11-20 08:47:30.848662", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYMy2Smc6KniqO9QcZBFSz0iD870sQcciE"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:30.707142", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138767 s, 831 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138767 s, 831 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:31.010 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJkMd4780TxMogesIHF6qK8SaQx7KT5XJx'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.151562", "end": "2021-11-20 08:47:30.978838", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJkMd4780TxMogesIHF6qK8SaQx7KT5XJx"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:30.827276", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14895 s, 774 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14895 s, 774 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:31.097 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-NQBaOY-ggvG-sSws-P3O2-7CzV-miF2-sz7411', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040150400BGN', 'nvme-nvme.8086-50484654363230343031353034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.038943", "end": "2021-11-20 08:47:31.068328", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-NQBaOY-ggvG-sSws-P3O2-7CzV-miF2-sz7411", "nvme-INTEL_SSDPEDMD400G4_PHFT62040150400BGN", "nvme-nvme.8086-50484654363230343031353034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:31.029385", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0362525 s, 3.2 GB/s", "std 2021-11-20T08:47:31.099 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0362525 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:31.215 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-mci9wg-ZWzb-PVEO-TcnL-8Oc1-TfUJ-LPKjT7', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN', 'nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039561", "end": "2021-11-20 08:47:31.192134", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-mci9wg-ZWzb-PVEO-TcnL-8Oc1-TfUJ-LPKjT7", "nvme-INTEL_SSDPEDMD400G4_PHFT620400VK400BGN", "nvme-nvme.8086-50484654363230343030564b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:31.152573", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.036883 s, 3.1 GB/s", "stde 2021-11-20T08:47:31.217 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.036883 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:31.415 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYOpSZlXnUdalZPcyG9dio6mNzVuMlZF0W'], 'uuids': ['efc7b32b-bdbf-4035-a570-287e69ca28ca'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140800", "end": "2021-11-20 08:47:31.390681", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYOpSZlXnUdalZPcyG9dio6mNzVuMlZF0W"], "labels": [], "masters": [], "uuids": ["efc7b32b-bdbf-4035-a570-287e69ca28ca"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:31.249881", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138217 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138217 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:31.541 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJz37z8y2TQJZGC44x2ND1Xm4WXCfO9N8W'], 'uuids': ['7db59f61-c64d-4278-8d4c-dc274028b4a8'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.142692", "end": "2021-11-20 08:47:31.517225", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJz37z8y2TQJZGC44x2ND1Xm4WXCfO9N8W"], "labels": [], "masters": [], "uuids": ["7db59f61-c64d-4278-8d4c-dc274028b4a8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:31.374533", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140033 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140033 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:31.730 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDY1fWsvvj8cyD3h2EN7agds9TmaNm2JyZg'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141659", "end": "2021-11-20 08:47:31.703907", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDY1fWsvvj8cyD3h2EN7agds9TmaNm2JyZg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:31.562248", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139072 s, 829 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139072 s, 829 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:31.858 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJdbqZdphZ0Q2Vjre9wuc1Y0aa5OP88S9E'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.144089", "end": "2021-11-20 08:47:31.833500", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJdbqZdphZ0Q2Vjre9wuc1Y0aa5OP88S9E"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:31.689411", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141413 s, 816 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141413 s, 816 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.048 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYihimHZCHNwSLUciYspfTKfw4lXuBZ4NK'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144222", "end": "2021-11-20 08:47:32.028815", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYihimHZCHNwSLUciYspfTKfw4lXuBZ4NK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:31.884593", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141542 s, 815 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141542 s, 815 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.060 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5Z8', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5Z8', 'scsi-35000c500918e738f', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8', 'wwn-0x5000c500918e738f'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5L5Z8', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918e738f', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5Z8-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5Z8-part1', 'scsi-35000c500918e738f-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1', 'wwn-0x5000c500918e738f-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5Z8", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5L5Z8", "scsi-35000c500918e738f", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8", "wwn-0x5000c500918e738f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5Z8-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T08:47:32.062 INFO:teuthology.task.ansible.out:Z1W5L5Z8-part1", "scsi-35000c500918e738f-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5L5Z8-part1", "wwn-0x5000c500918e738f-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L5Z8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e738f"}}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:32.176 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJHRScsnJdlK0ljA9tT3kDCem0d7T8OqYj'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144631", "end": "2021-11-20 08:47:32.149909", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJHRScsnJdlK0ljA9tT3kDCem0d7T8OqYj"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:32.005278", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141935 s, 813 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141935 s, 813 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.177 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM', 'scsi-35000c50091e0bcf8', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM', 'wwn-0x5000c50091e0bcf8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QPPM', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0bcf8', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM-part1', 'scsi-35000c50091e0bcf8-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1', 'wwn-0x5000c50091e0bcf8-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPPM", "scsi-35000c50091e0bcf8", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM", "wwn-0x5000c50091e0bcf8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPPM-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T08:47:32.179 INFO:teuthology.task.ansible.out:Z1W5QPPM-part1", "scsi-35000c50091e0bcf8-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPPM-part1", "wwn-0x5000c50091e0bcf8-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QPPM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0bcf8"}}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:32.378 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYq1OrsVIJgECpoif1Xk5ZR4vOqa3dvYz8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.151333", "end": "2021-11-20 08:47:32.361317", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PWQ8AnQhOYhJ4HHZYpLyFghI5tM9KoDYq1OrsVIJgECpoif1Xk5ZR4vOqa3dvYz8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:32.209984", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148697 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148697 s, 776 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.419 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:32 +0000 (0:00:01.793) 0:05:06.364 ***** 2021-11-20T08:47:32.500 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJ8GGziliEt4VCHHz8FDcf0wTKTwjmD4tf'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.143488", "end": "2021-11-20 08:47:32.470179", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HxVKGCJ8CewHeLQHL24Ywwonsm3j88aJ8GGziliEt4VCHHz8FDcf0wTKTwjmD4tf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 08:47:32.326691", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140826 s, 819 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140826 s, 819 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.532 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:32 +0000 (0:00:00.113) 0:05:06.477 ***** 2021-11-20T08:47:32.654 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-20T08:47:32.655 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.047505", "end": "2021-11-20 08:47:32.627272", "rc": 0, "start": "2021-11-20 08:47:32.579767", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.679 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:32 +0000 (0:00:00.145) 0:05:06.623 ***** 2021-11-20T08:47:32.769 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.052369", "end": "2021-11-20 08:47:32.746030", "rc": 0, "start": "2021-11-20 08:47:32.693661", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.795 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:32 +0000 (0:00:00.116) 0:05:06.740 ***** 2021-11-20T08:47:32.914 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049564", "end": "2021-11-20 08:47:32.890335", "rc": 0, "start": "2021-11-20 08:47:32.840771", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:32.940 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:32 +0000 (0:00:00.145) 0:05:06.885 ***** 2021-11-20T08:47:32.969 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T08:47:32.995 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:32 +0000 (0:00:00.055) 0:05:06.940 ***** 2021-11-20T08:47:33.044 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-20T08:47:33.046 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.062214", "end": "2021-11-20 08:47:33.026303", "rc": 0, "start": "2021-11-20 08:47:32.964089", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:33.210 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.214) 0:05:07.155 ***** 2021-11-20T08:47:33.228 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-20T08:47:33.230 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.253 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.043) 0:05:07.199 ***** 2021-11-20T08:47:33.261 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T08:47:33.288 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.034) 0:05:07.233 ***** 2021-11-20T08:47:33.296 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.321 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.032) 0:05:07.266 ***** 2021-11-20T08:47:33.330 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.358 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.036) 0:05:07.303 ***** 2021-11-20T08:47:33.365 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.392 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.034) 0:05:07.337 ***** 2021-11-20T08:47:33.400 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-20T08:47:33.402 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.425 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.033) 0:05:07.371 ***** 2021-11-20T08:47:33.432 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T08:47:33.434 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.458 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.032) 0:05:07.404 ***** 2021-11-20T08:47:33.468 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T08:47:33.471 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.502 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.043) 0:05:07.447 ***** 2021-11-20T08:47:33.531 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T08:47:33.533 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.556 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.054) 0:05:07.501 ***** 2021-11-20T08:47:33.661 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T08:47:33.663 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:47:33.689 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.132) 0:05:07.634 ***** 2021-11-20T08:47:33.742 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T08:47:33.745 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140160000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085000, 'block_used': 13010108, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085000, "block_size": 4096, "block_total": 240095108, "block_used": 13010108, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140160000, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.779 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.089) 0:05:07.724 ***** 2021-11-20T08:47:33.793 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T08:47:33.795 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T08:47:33.828 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.049) 0:05:07.774 ***** 2021-11-20T08:47:33.835 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T08:47:33.836 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.860 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.031) 0:05:07.805 ***** 2021-11-20T08:47:33.882 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi102.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140184576, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085006, 'block_used': 13010102, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085006, "block_size": 4096, "block_total": 240095108, "block_used": 13010102, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140184576, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.913 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.053) 0:05:07.858 ***** 2021-11-20T08:47:33.923 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:47:33.926 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.958 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.044) 0:05:07.902 ***** 2021-11-20T08:47:33.964 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T08:47:33.966 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:33.991 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:33 +0000 (0:00:00.033) 0:05:07.936 ***** 2021-11-20T08:47:34.003 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.026 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.033) 0:05:07.970 ***** 2021-11-20T08:47:34.034 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.040) 0:05:08.011 ***** 2021-11-20T08:47:34.076 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-11-20T08:47:34.078 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.101 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.036) 0:05:08.047 ***** 2021-11-20T08:47:34.108 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:47:34.114 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.138 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.033) 0:05:08.080 ***** 2021-11-20T08:47:34.155 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.177 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.041) 0:05:08.122 ***** 2021-11-20T08:47:34.185 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.210 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.033) 0:05:08.156 ***** 2021-11-20T08:47:34.230 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.252 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.041) 0:05:08.197 ***** 2021-11-20T08:47:34.259 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-20T08:47:34.261 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.284 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.032) 0:05:08.230 ***** 2021-11-20T08:47:34.313 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-20T08:47:34.314 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:34.336 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.051) 0:05:08.281 ***** 2021-11-20T08:47:34.772 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-11-20T08:47:34.774 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-20T08:47:34.796 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.460) 0:05:08.742 ***** 2021-11-20T08:47:34.818 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:34 +0000 (0:00:00.021) 0:05:08.764 ***** 2021-11-20T08:47:35.266 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T08:47:35.268 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:35.288 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:35.593 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:35.625 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:35.950 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:36.003 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:36.298 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:36.351 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:36.646 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:36.676 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:36 +0000 (0:00:01.857) 0:05:10.622 ***** 2021-11-20T08:47:36.699 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T08:47:36.724 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:36 +0000 (0:00:00.048) 0:05:10.670 ***** 2021-11-20T08:47:36.882 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T08:47:36.910 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:36 +0000 (0:00:00.183) 0:05:10.854 ***** 2021-11-20T08:47:36.928 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T08:47:36.955 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:36 +0000 (0:00:00.046) 0:05:10.900 ***** 2021-11-20T08:47:37.127 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T08:47:37.129 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.173 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.314 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.347 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.509 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.545 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.694 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.716 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:37.745 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:37 +0000 (0:00:00.788) 0:05:11.689 ***** 2021-11-20T08:47:37.754 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T08:47:37.756 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:37.795 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:37 +0000 (0:00:00.050) 0:05:11.740 ***** 2021-11-20T08:47:37.806 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-11-20T08:47:37.808 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:37.845 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:37 +0000 (0:00:00.050) 0:05:11.790 ***** 2021-11-20T08:47:37.856 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:37.881 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:37 +0000 (0:00:00.035) 0:05:11.826 ***** 2021-11-20T08:47:37.889 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:37.911 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:37 +0000 (0:00:00.030) 0:05:11.856 ***** 2021-11-20T08:47:37.918 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:37.943 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:37 +0000 (0:00:00.031) 0:05:11.888 ***** 2021-11-20T08:47:38.122 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:47:38.152 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:38 +0000 (0:00:00.208) 0:05:12.096 ***** 2021-11-20T08:47:38.162 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:47:38.185 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:38 +0000 (0:00:00.034) 0:05:12.131 ***** 2021-11-20T08:47:38.187 INFO:teuthology.task.ansible.out: 2021-11-20T08:47:38.193 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-11-20T08:47:38.197 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T08:47:38.220 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:38 +0000 (0:00:00.035) 0:05:12.166 ***** 2021-11-20T08:47:38.228 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T08:47:38.253 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:38 +0000 (0:00:00.032) 0:05:12.199 ***** 2021-11-20T08:47:38.599 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi093.front.sepia.ceph.com] => {"changed": true} 2021-11-20T08:47:38.626 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:38 +0000 (0:00:00.372) 0:05:12.571 ***** 2021-11-20T08:47:38.635 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true} 2021-11-20T08:47:38.662 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:38 +0000 (0:00:00.036) 0:05:12.607 ***** 2021-11-20T08:47:39.036 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-20T08:47:39.038 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-20T08:47:39.059 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.397) 0:05:13.005 ***** 2021-11-20T08:47:39.082 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.022) 0:05:13.027 ***** 2021-11-20T08:47:39.100 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-11-20T08:47:39.102 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.123 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.041) 0:05:13.069 ***** 2021-11-20T08:47:39.131 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.155 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.031) 0:05:13.100 ***** 2021-11-20T08:47:39.162 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.186 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.031) 0:05:13.131 ***** 2021-11-20T08:47:39.194 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.224 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.038) 0:05:13.170 ***** 2021-11-20T08:47:39.232 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-20T08:47:39.233 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.261 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.034) 0:05:13.204 ***** 2021-11-20T08:47:39.267 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.291 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.032) 0:05:13.237 ***** 2021-11-20T08:47:39.300 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-20T08:47:39.302 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.325 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.033) 0:05:13.271 ***** 2021-11-20T08:47:39.332 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.356 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.030) 0:05:13.301 ***** 2021-11-20T08:47:39.364 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-11-20T08:47:39.366 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.389 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.033) 0:05:13.334 ***** 2021-11-20T08:47:39.397 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.422 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.032) 0:05:13.367 ***** 2021-11-20T08:47:39.424 INFO:teuthology.task.ansible.out: 2021-11-20T08:47:39.433 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.457 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.034) 0:05:13.402 ***** 2021-11-20T08:47:39.465 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.489 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.031) 0:05:13.434 ***** 2021-11-20T08:47:39.496 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-11-20T08:47:39.497 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.526 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.037) 0:05:13.472 ***** 2021-11-20T08:47:39.534 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.559 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.032) 0:05:13.504 ***** 2021-11-20T08:47:39.572 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-11-20T08:47:39.574 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.611 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.052) 0:05:13.557 ***** 2021-11-20T08:47:39.622 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.647 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.035) 0:05:13.592 ***** 2021-11-20T08:47:39.655 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-11-20T08:47:39.658 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.681 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.034) 0:05:13.626 ***** 2021-11-20T08:47:39.690 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.714 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.032) 0:05:13.659 ***** 2021-11-20T08:47:39.721 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-11-20T08:47:39.723 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.751 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.034) 0:05:13.694 ***** 2021-11-20T08:47:39.755 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.779 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.030) 0:05:13.724 ***** 2021-11-20T08:47:39.785 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-11-20T08:47:39.787 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.810 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.031) 0:05:13.755 ***** 2021-11-20T08:47:39.818 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.841 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.031) 0:05:13.787 ***** 2021-11-20T08:47:39.848 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-11-20T08:47:39.850 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.872 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.030) 0:05:13.818 ***** 2021-11-20T08:47:39.880 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.905 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.030) 0:05:13.848 ***** 2021-11-20T08:47:39.909 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-11-20T08:47:39.911 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.935 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.031) 0:05:13.880 ***** 2021-11-20T08:47:39.946 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:39.974 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:39 +0000 (0:00:00.038) 0:05:13.919 ***** 2021-11-20T08:47:39.982 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-11-20T08:47:39.984 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:40.011 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.037) 0:05:13.956 ***** 2021-11-20T08:47:40.019 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:40.042 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.031) 0:05:13.987 ***** 2021-11-20T08:47:40.050 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-11-20T08:47:40.052 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:40.078 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.035) 0:05:14.023 ***** 2021-11-20T08:47:40.087 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:40.111 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.032) 0:05:14.056 ***** 2021-11-20T08:47:40.570 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T08:47:40.596 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.485) 0:05:14.542 ***** 2021-11-20T08:47:40.604 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T08:47:40.628 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.032) 0:05:14.574 ***** 2021-11-20T08:47:40.898 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-11-20T08:47:40.899 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 08:41:17 UTC", "ActiveEnterTimestampMonotonic": "12873441", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service basic.target -.mount systemd-tmpfiles-setup.service systemd-journald.socket sysinit.target tmp.mount ntpd.service system.slice sntp.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 08:41:16 UTC", "AssertTimestampMonotonic": "11566608", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 08:41:16 UTC", "ConditionTimestampMonotonic": "11566565", "ConfigurationDirectoryMode": "0755", "Conflicts": "syst 2021-11-20T08:47:40.901 INFO:teuthology.task.ansible.out:emd-timesyncd.service shutdown.target ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "877", "ExecMainStartTimestamp": "Sat 2021-11-20 08:41:17 UTC", "ExecMainStartTimestampMonotonic": "12805102", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 08:41:16 UTC", "InactiveExitTimestampMonotonic": "11591989", "InvocationID": "1c49a40930cc4e62b65180a60da30430", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T08:47:40.902 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127241", "LimitNPROCSoft": "127241", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127241", "LimitSIGPENDINGSoft": "127241", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "877", "MemoryAccounting": "yes", "MemoryCurrent": "1708032", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice 2021-11-20T08:47:40.904 INFO:teuthology.task.ansible.out:-.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 08:41:17 UTC", "StateChangeTimestampMonotonic": "12873441", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203586", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 08:41:17 UTC", "WatchdogTimestampMonotonic": "12805113", "WatchdogUSec": "0"}} 2021-11-20T08:47:40.923 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.294) 0:05:14.868 ***** 2021-11-20T08:47:40.933 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 08:41:15 UTC", "ActiveEnterTimestampMonotonic": "12006345", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount ntpd.service system.slice systemd-journald.socket basic.target systemd-tmpfiles-setup.service ntpdate.service tmp.mount sntp.service sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 08:41:13 UTC", "AssertTimestampMonotonic": "10279511", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 08:41:13 UTC", "ConditionTimestampMonotonic": "10279470", "ConfigurationDirectoryMode": "0755", "Conflicts": "syst 2021-11-20T08:47:40.934 INFO:teuthology.task.ansible.out:emd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "836", "ExecMainStartTimestamp": "Sat 2021-11-20 08:41:15 UTC", "ExecMainStartTimestampMonotonic": "11933248", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 08:41:13 UTC", "InactiveExitTimestampMonotonic": "10324351", "InvocationID": "5b91fda9cc8d4518b0d51ad2f607747c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T08:47:40.936 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "836", "MemoryAccounting": "yes", "MemoryCurrent": "1966080", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit 2021-11-20T08:47:40.936 INFO:teuthology.task.ansible.out:.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 08:41:15 UTC", "StateChangeTimestampMonotonic": "12006345", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 08:41:15 UTC", "WatchdogTimestampMonotonic": "11933281", "WatchdogUSec": "0"}} 2021-11-20T08:47:40.956 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.032) 0:05:14.901 ***** 2021-11-20T08:47:40.962 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:47:40.964 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T08:47:40.986 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:40 +0000 (0:00:00.030) 0:05:14.932 ***** 2021-11-20T08:47:40.993 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T08:47:41.018 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.031) 0:05:14.963 ***** 2021-11-20T08:47:41.034 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T08:47:41.036 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T08:47:41.057 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.039) 0:05:15.003 ***** 2021-11-20T08:47:41.066 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T08:47:41.094 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.036) 0:05:15.039 ***** 2021-11-20T08:47:41.101 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-11-20T08:47:41.103 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:41.125 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.031) 0:05:15.071 ***** 2021-11-20T08:47:41.134 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:41.158 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.032) 0:05:15.104 ***** 2021-11-20T08:47:41.166 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-11-20T08:47:41.168 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:41.190 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.032) 0:05:15.136 ***** 2021-11-20T08:47:41.199 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:41.222 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.031) 0:05:15.168 ***** 2021-11-20T08:47:41.230 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-11-20T08:47:41.231 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:41.253 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.030) 0:05:15.199 ***** 2021-11-20T08:47:41.260 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:41.285 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.031) 0:05:15.230 ***** 2021-11-20T08:47:41.449 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-11-20T08:47:41.451 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T08:47:41.474 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.188) 0:05:15.419 ***** 2021-11-20T08:47:41.492 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T08:47:41.515 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.041) 0:05:15.461 ***** 2021-11-20T08:47:41.950 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-11-20T08:47:41.951 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T08:47:41.973 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:41 +0000 (0:00:00.457) 0:05:15.919 ***** 2021-11-20T08:47:42.002 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T08:47:42.025 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:42 +0000 (0:00:00.051) 0:05:15.970 ***** 2021-11-20T08:47:43.101 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T08:47:43.132 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:01.107) 0:05:17.078 ***** 2021-11-20T08:47:43.145 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T08:47:43.169 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.036) 0:05:17.114 ***** 2021-11-20T08:47:43.177 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:43.203 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.033) 0:05:17.147 ***** 2021-11-20T08:47:43.210 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:43.234 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.032) 0:05:17.180 ***** 2021-11-20T08:47:43.242 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-11-20T08:47:43.244 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-20T08:47:43.269 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.034) 0:05:17.214 ***** 2021-11-20T08:47:43.277 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-20T08:47:43.301 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.032) 0:05:17.246 ***** 2021-11-20T08:47:43.458 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T08:47:43.482 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.180) 0:05:17.427 ***** 2021-11-20T08:47:43.492 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T08:47:43.517 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.034) 0:05:17.462 ***** 2021-11-20T08:47:43.524 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-11-20T08:47:43.525 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:43.550 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.033) 0:05:17.496 ***** 2021-11-20T08:47:43.558 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:43.589 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.039) 0:05:17.535 ***** 2021-11-20T08:47:43.597 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-11-20T08:47:43.599 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-20T08:47:43.624 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.032) 0:05:17.567 ***** 2021-11-20T08:47:43.629 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-20T08:47:43.653 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.031) 0:05:17.599 ***** 2021-11-20T08:47:43.659 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-11-20T08:47:43.661 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:43.683 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.029) 0:05:17.628 ***** 2021-11-20T08:47:43.689 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:43.714 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.031) 0:05:17.659 ***** 2021-11-20T08:47:43.880 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-11-20T08:47:43.882 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T08:47:43.905 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.190) 0:05:17.850 ***** 2021-11-20T08:47:43.913 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T08:47:43.939 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:43 +0000 (0:00:00.034) 0:05:17.885 ***** 2021-11-20T08:47:44.106 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi093.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T08:47:44.129 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.189) 0:05:18.074 ***** 2021-11-20T08:47:44.149 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T08:47:44.172 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.043) 0:05:18.118 ***** 2021-11-20T08:47:44.179 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-11-20T08:47:44.181 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.203 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.030) 0:05:18.149 ***** 2021-11-20T08:47:44.211 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.237 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.032) 0:05:18.182 ***** 2021-11-20T08:47:44.254 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-11-20T08:47:44.256 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.267 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.292 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.055) 0:05:18.237 ***** 2021-11-20T08:47:44.299 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.301 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.301 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.302 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.327 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.035) 0:05:18.273 ***** 2021-11-20T08:47:44.335 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-11-20T08:47:44.338 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.348 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.350 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi093.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.377 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.049) 0:05:18.322 ***** 2021-11-20T08:47:44.388 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.389 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.401 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi102.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.403 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.430 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.053) 0:05:18.376 ***** 2021-11-20T08:47:44.451 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.020) 0:05:18.396 ***** 2021-11-20T08:47:44.457 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-11-20T08:47:44.459 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.482 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.031) 0:05:18.428 ***** 2021-11-20T08:47:44.491 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.516 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.033) 0:05:18.461 ***** 2021-11-20T08:47:44.522 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-11-20T08:47:44.524 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.547 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.031) 0:05:18.493 ***** 2021-11-20T08:47:44.556 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-11-20T08:47:44.558 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.030) 0:05:18.524 ***** 2021-11-20T08:47:44.585 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-20T08:47:44.587 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.609 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.031) 0:05:18.555 ***** 2021-11-20T08:47:44.616 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.641 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.031) 0:05:18.586 ***** 2021-11-20T08:47:44.647 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-11-20T08:47:44.648 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.675 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.034) 0:05:18.620 ***** 2021-11-20T08:47:44.682 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-20T08:47:44.684 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.706 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.030) 0:05:18.651 ***** 2021-11-20T08:47:44.712 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T08:47:44.714 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.740 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.034) 0:05:18.686 ***** 2021-11-20T08:47:44.747 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-11-20T08:47:44.749 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.774 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.034) 0:05:18.720 ***** 2021-11-20T08:47:44.781 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-11-20T08:47:44.782 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.809 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.034) 0:05:18.754 ***** 2021-11-20T08:47:44.819 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T08:47:44.821 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.846 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.037) 0:05:18.791 ***** 2021-11-20T08:47:44.854 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.880 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.033) 0:05:18.825 ***** 2021-11-20T08:47:44.889 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.916 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.036) 0:05:18.862 ***** 2021-11-20T08:47:44.945 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:44.968 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:44 +0000 (0:00:00.051) 0:05:18.914 ***** 2021-11-20T08:47:45.091 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-20T08:47:45.092 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005236", "end": "2021-11-20 08:47:45.062586", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 08:47:45.057350", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:45.186 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005184", "end": "2021-11-20 08:47:45.156752", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 08:47:45.151568", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:45.272 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003227", "end": "2021-11-20 08:47:45.243620", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 08:47:45.240393", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:45.365 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003082", "end": "2021-11-20 08:47:45.335039", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 08:47:45.331957", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:45.450 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002944", "end": "2021-11-20 08:47:45.418635", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 08:47:45.415691", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:45.545 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002861", "end": "2021-11-20 08:47:45.513711", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 08:47:45.510850", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:45.620 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003096", "end": "2021-11-20 08:47:45.592914", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 08:47:45.589818", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:47:45.785 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:45 +0000 (0:00:00.681) 0:05:19.595 ***** TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** Saturday 20 November 2021 08:47:45 +0000 (0:00:00.051) 0:05:19.647 ***** TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi102.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003093", "end": "2021-11-20 08:47:45.688253", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 08:47:45.685160", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Saturday 20 November 2021 08:47:45 +0000 (0:00:00.046) 0:05:19.693 ***** TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:45.799 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-20T08:47:45.819 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:45 +0000 (0:00:00.071) 0:05:19.765 ***** 2021-11-20T08:47:45.821 INFO:teuthology.task.ansible.out: 2021-11-20T08:47:45.847 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:45.915 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-11-20T08:47:45.934 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:45 +0000 (0:00:00.114) 0:05:19.879 ***** 2021-11-20T08:47:45.955 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:45 +0000 (0:00:00.021) 0:05:19.900 ***** 2021-11-20T08:47:46.572 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi093.front.sepia.ceph.com] 2021-11-20T08:47:46.604 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-11-20T08:47:46.631 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.676) 0:05:20.577 ***** 2021-11-20T08:47:46.676 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.044) 0:05:20.621 ***** 2021-11-20T08:47:46.682 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T08:47:46.684 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:46.706 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.030) 0:05:20.652 ***** 2021-11-20T08:47:46.714 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:47:46.738 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.032) 0:05:20.684 ***** 2021-11-20T08:47:46.744 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T08:47:46.746 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T08:47:46.769 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.030) 0:05:20.715 ***** 2021-11-20T08:47:46.776 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T08:47:46.800 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.030) 0:05:20.745 ***** 2021-11-20T08:47:46.818 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-11-20T08:47:46.819 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T08:47:46.841 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.041) 0:05:20.787 ***** 2021-11-20T08:47:46.849 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T08:47:46.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:47:46 +0000 (0:00:00.032) 0:05:20.820 ***** 2021-11-20T08:48:29.106 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T08:48:29.108 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T08:48:29.129 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:29 +0000 (0:00:42.254) 0:06:03.074 ***** 2021-11-20T08:48:29.166 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T08:48:29.168 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T08:48:29.191 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:29 +0000 (0:00:00.061) 0:06:03.136 ***** 2021-11-20T08:48:29.218 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:29.241 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:29 +0000 (0:00:00.050) 0:06:03.186 ***** 2021-11-20T08:48:29.439 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T08:48:29.441 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T08:48:29.465 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:29 +0000 (0:00:00.224) 0:06:03.411 ***** 2021-11-20T08:48:29.478 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T08:48:29.479 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:48:29.502 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi093.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T08:48:29.526 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:29 +0000 (0:00:00.060) 0:06:03.471 ***** 2021-11-20T08:48:29.553 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:29.576 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:29 +0000 (0:00:00.050) 0:06:03.521 ***** 2021-11-20T08:48:29.656 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T08:48:29.658 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:48:29.683 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:29 +0000 (0:00:00.107) 0:06:03.629 ***** 2021-11-20T08:48:29.796 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:48:29.985 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T08:48:30.010 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:30 +0000 (0:00:00.326) 0:06:03.956 ***** 2021-11-20T08:48:30.225 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-11-20T08:48:30.226 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637398109.755351-15966-153454006544442/source", "state": "file", "uid": 0} 2021-11-20T08:48:30.570 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637398110.0971544-16008-62886744168042/source", "state": "file", "uid": 0} 2021-11-20T08:48:30.728 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637398110.2630074-15966-268912458865283/source", "state": "file", "uid": 0} 2021-11-20T08:48:30.757 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:30 +0000 (0:00:00.746) 0:06:04.703 ***** 2021-11-20T08:48:31.051 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637398110.600958-16008-17053467959061/source", "state": "file", "uid": 0} 2021-11-20T08:48:31.078 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:31 +0000 (0:00:00.320) 0:06:05.023 ***** 2021-11-20T08:48:33.009 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-11-20T08:48:33.011 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:48:33.043 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:33 +0000 (0:00:01.965) 0:06:06.989 ***** 2021-11-20T08:48:33.072 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T08:48:33.098 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:33 +0000 (0:00:00.055) 0:06:07.044 ***** 2021-11-20T08:48:36.396 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-20T08:48:36.398 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-k4ukxx3d-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-k4ukxx3d-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-20T08:48:36.400 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-3ei6qs4i-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with 2021-11-20T08:48:36.401 INFO:teuthology.task.ansible.out:status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-3ei6qs4i-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-20T08:48:36.420 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:03.321) 0:06:10.365 ***** 2021-11-20T08:48:36.442 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:00.022) 0:06:10.388 ***** 2021-11-20T08:48:36.451 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-20T08:48:36.454 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:36.481 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:00.038) 0:06:10.426 ***** 2021-11-20T08:48:36.490 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:36.514 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:00.032) 0:06:10.459 ***** 2021-11-20T08:48:36.522 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:36.547 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:00.033) 0:06:10.493 ***** 2021-11-20T08:48:36.559 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:36.586 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:00.038) 0:06:10.532 ***** 2021-11-20T08:48:36.895 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-20T08:48:36.896 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.147133", "end": "2021-11-20 08:48:36.873765", "rc": 0, "start": "2021-11-20 08:48:36.726632", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:48:36.918 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:00.332) 0:06:10.864 ***** 2021-11-20T08:48:36.938 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.152993", "end": "2021-11-20 08:48:36.914614", "rc": 0, "start": "2021-11-20 08:48:36.761621", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T08:48:36.960 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:36 +0000 (0:00:00.041) 0:06:10.905 ***** 2021-11-20T08:48:36.967 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-11-20T08:48:36.969 INFO:teuthology.task.ansible.out:skipping: [smithi093.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:36.994 INFO:teuthology.task.ansible.out:skipping: [smithi102.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T08:48:37.075 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-11-20T08:48:37.076 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-11-20T08:48:37.077 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-11-20T08:48:37.079 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T08:48:37.092 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 20 November 2021 08:48:37 +0000 (0:00:00.132) 0:06:11.038 ***** 2021-11-20T08:48:37.743 INFO:teuthology.task.ansible.out:ok: [smithi093.front.sepia.ceph.com] 2021-11-20T08:48:37.745 INFO:teuthology.task.ansible.out: 2021-11-20T08:48:38.786 INFO:teuthology.task.ansible.out:ok: [smithi102.front.sepia.ceph.com] 2021-11-20T08:48:38.802 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** 2021-11-20T08:48:38.804 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 08:48:38 +0000 (0:00:01.710) 0:06:12.748 ***** 2021-11-20T08:48:39.030 INFO:teuthology.task.ansible.out:changed: [smithi093.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T08:48:39.053 INFO:teuthology.task.ansible.out:changed: [smithi102.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T08:48:39.055 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi093.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 2021-11-20T08:48:39.057 INFO:teuthology.task.ansible.out:smithi102.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 Saturday 20 November 2021 08:48:39 +0000 (0:00:00.252) 0:06:13.001 ***** =============================================================================== Create all admin users with sudo access. ------------------------------- 59.88s container-host : Install container packages ---------------------------- 42.25s 2021-11-20T08:48:39.057 INFO:teuthology.task.ansible.out:common : nrpe - Ensure SELinux policy is up to date -------------------- 39.69s users : Update authorized_keys using the keys repo --------------------- 31.65s Remove revoked users --------------------------------------------------- 18.22s testnode : Ensure ceph packages are not present. ----------------------- 17.96s common : nrpe - Remove SELinux policy package -------------------------- 15.73s common : nrpe - Load SELinux policy package ---------------------------- 15.65s testnode : Install packages --------------------------------------------- 9.02s testnode : Zap all non-root disks --------------------------------------- 7.98s testnode : Upgrade packages --------------------------------------------- 7.69s testnode : Install GPG -------------------------------------------------- 6.97s common : Install Kerberos Packages (RedHat) ----------------------------- 4.38s container-host : Install registries-conf-ctl ---------------------------- 3.32s testnode : Ensure ceph dependency packages are not present. ------------- 2.87s testnode : Install GPG keys --------------------------------------------- 2.44s testnode : Rebuild rpmdb ------------------------------------------------ 2.34s testnode : Remove packages ---------------------------------------------- 2.11s container-host : Ensure git is installed -------------------------------- 1.97s testnode : Make sure rpm dependencies are installed --------------------- 1.94s 2021-11-20T08:48:39.507 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi093.front.sepia.ceph.com'), Remote(name='ubuntu@smithi102.front.sepia.ceph.com')] 2021-11-20T08:48:39.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2021-11-20T08:48:39.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:48:39.586 DEBUG:teuthology.orchestra.run.smithi093:> true 2021-11-20T08:48:39.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2021-11-20T08:48:39.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T08:48:39.726 DEBUG:teuthology.orchestra.run.smithi102:> true 2021-11-20T08:48:39.792 INFO:teuthology.run_tasks:Running task clock... 2021-11-20T08:48:39.803 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-11-20T08:48:39.804 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T08:48:39.806 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T08:48:39.808 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T08:48:39.809 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T08:48:39.837 INFO:teuthology.orchestra.run.smithi093.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T08:48:39.852 INFO:teuthology.orchestra.run.smithi093.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T08:48:39.863 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T08:48:39.877 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T08:48:39.886 INFO:teuthology.orchestra.run.smithi093.stderr:sudo: ntpd: command not found 2021-11-20T08:48:39.896 INFO:teuthology.orchestra.run.smithi093.stdout:506 Cannot talk to daemon 2021-11-20T08:48:39.910 INFO:teuthology.orchestra.run.smithi093.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T08:48:39.920 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2021-11-20T08:48:39.923 INFO:teuthology.orchestra.run.smithi093.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T08:48:39.930 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2021-11-20T08:48:39.944 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T08:48:39.958 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T08:48:39.988 INFO:teuthology.orchestra.run.smithi093.stderr:bash: ntpq: command not found 2021-11-20T08:48:39.990 INFO:teuthology.orchestra.run.smithi093.stdout:210 Number of sources = 4 2021-11-20T08:48:39.990 INFO:teuthology.orchestra.run.smithi093.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T08:48:39.990 INFO:teuthology.orchestra.run.smithi093.stdout:=============================================================================== 2021-11-20T08:48:39.990 INFO:teuthology.orchestra.run.smithi093.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:39.991 INFO:teuthology.orchestra.run.smithi093.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:39.991 INFO:teuthology.orchestra.run.smithi093.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:39.991 INFO:teuthology.orchestra.run.smithi093.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2021-11-20T08:48:40.033 INFO:teuthology.orchestra.run.smithi102.stdout:210 Number of sources = 4 2021-11-20T08:48:40.034 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T08:48:40.034 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2021-11-20T08:48:40.034 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:40.034 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:40.035 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:40.035 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T08:48:40.036 INFO:teuthology.run_tasks:Running task pexec... 2021-11-20T08:48:40.052 INFO:teuthology.task.pexec:Executing custom commands... 2021-11-20T08:48:40.053 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T08:48:40.054 DEBUG:teuthology.orchestra.run.smithi093:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T08:48:40.055 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi102.front.sepia.ceph.com 2021-11-20T08:48:40.055 DEBUG:teuthology.orchestra.run.smithi102:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T08:48:40.619 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:54 ago on Sat 20 Nov 2021 08:46:46 AM UTC. 2021-11-20T08:48:40.641 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:53 ago on Sat 20 Nov 2021 08:46:47 AM UTC. 2021-11-20T08:48:42.065 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:48:42.066 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:48:42.067 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:48:42.067 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:48:42.067 INFO:teuthology.orchestra.run.smithi102.stdout:Resetting modules: 2021-11-20T08:48:42.067 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools 2021-11-20T08:48:42.068 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:48:42.068 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:48:42.068 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:48:42.068 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:48:42.093 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:48:42.093 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:48:42.094 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:48:42.094 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:48:42.094 INFO:teuthology.orchestra.run.smithi093.stdout:Resetting modules: 2021-11-20T08:48:42.095 INFO:teuthology.orchestra.run.smithi093.stdout: container-tools 2021-11-20T08:48:42.095 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:48:42.095 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:48:42.095 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:48:42.096 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:48:42.291 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:48:42.318 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:48:42.891 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 08:46:47 AM UTC. 2021-11-20T08:48:42.941 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:56 ago on Sat 20 Nov 2021 08:46:46 AM UTC. 2021-11-20T08:48:44.348 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:48:44.479 INFO:teuthology.orchestra.run.smithi093.stdout:===================================================================================================== 2021-11-20T08:48:44.480 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:48:44.480 INFO:teuthology.orchestra.run.smithi093.stdout:===================================================================================================== 2021-11-20T08:48:44.480 INFO:teuthology.orchestra.run.smithi093.stdout:Upgrading: 2021-11-20T08:48:44.480 INFO:teuthology.orchestra.run.smithi093.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T08:48:44.481 INFO:teuthology.orchestra.run.smithi093.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T08:48:44.481 INFO:teuthology.orchestra.run.smithi093.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T08:48:44.481 INFO:teuthology.orchestra.run.smithi093.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T08:48:44.481 INFO:teuthology.orchestra.run.smithi093.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T08:48:44.481 INFO:teuthology.orchestra.run.smithi093.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T08:48:44.482 INFO:teuthology.orchestra.run.smithi093.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T08:48:44.482 INFO:teuthology.orchestra.run.smithi093.stdout:Installing group/module packages: 2021-11-20T08:48:44.482 INFO:teuthology.orchestra.run.smithi093.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T08:48:44.482 INFO:teuthology.orchestra.run.smithi093.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T08:48:44.483 INFO:teuthology.orchestra.run.smithi093.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T08:48:44.483 INFO:teuthology.orchestra.run.smithi093.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T08:48:44.483 INFO:teuthology.orchestra.run.smithi093.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T08:48:44.483 INFO:teuthology.orchestra.run.smithi093.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T08:48:44.483 INFO:teuthology.orchestra.run.smithi093.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T08:48:44.484 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:48:44.484 INFO:teuthology.orchestra.run.smithi093.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T08:48:44.484 INFO:teuthology.orchestra.run.smithi093.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T08:48:44.484 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T08:48:44.485 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T08:48:44.485 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T08:48:44.485 INFO:teuthology.orchestra.run.smithi093.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T08:48:44.485 INFO:teuthology.orchestra.run.smithi093.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T08:48:44.485 INFO:teuthology.orchestra.run.smithi093.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T08:48:44.486 INFO:teuthology.orchestra.run.smithi093.stdout:Installing module profiles: 2021-11-20T08:48:44.486 INFO:teuthology.orchestra.run.smithi093.stdout: container-tools/common 2021-11-20T08:48:44.486 INFO:teuthology.orchestra.run.smithi093.stdout:Enabling module streams: 2021-11-20T08:48:44.488 INFO:teuthology.orchestra.run.smithi093.stdout: container-tools rhel8 2021-11-20T08:48:44.489 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:48:44.490 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:48:44.491 INFO:teuthology.orchestra.run.smithi093.stdout:===================================================================================================== 2021-11-20T08:48:44.492 INFO:teuthology.orchestra.run.smithi093.stdout:Install 15 Packages 2021-11-20T08:48:44.493 INFO:teuthology.orchestra.run.smithi093.stdout:Upgrade 7 Packages 2021-11-20T08:48:44.494 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:48:44.496 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 41 M 2021-11-20T08:48:44.496 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:48:44.496 INFO:teuthology.orchestra.run.smithi093.stdout:(1/22): cockpit-podman-35-1.module_el8.6.0+944+ 12 MB/s | 457 kB 00:00 2021-11-20T08:48:44.496 INFO:teuthology.orchestra.run.smithi093.stdout:(2/22): crun-1.2-1.module_el8.6.0+954+963caf36. 4.8 MB/s | 193 kB 00:00 2021-11-20T08:48:44.497 INFO:teuthology.orchestra.run.smithi093.stdout:(3/22): python3-podman-3.2.1-4.module_el8.6.0+9 43 MB/s | 148 kB 00:00 2021-11-20T08:48:44.497 INFO:teuthology.orchestra.run.smithi093.stdout:(4/22): python3-pytoml-0.1.14-5.git7dea353.el8. 16 MB/s | 25 kB 00:00 2021-11-20T08:48:44.497 INFO:teuthology.orchestra.run.smithi093.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 50 MB/s | 94 kB 00:00 2021-11-20T08:48:44.497 INFO:teuthology.orchestra.run.smithi093.stdout:(6/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 39 MB/s | 2.3 MB 00:00 2021-11-20T08:48:44.498 INFO:teuthology.orchestra.run.smithi093.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 14 MB/s | 48 kB 00:00 2021-11-20T08:48:44.498 INFO:teuthology.orchestra.run.smithi093.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 13 MB/s | 41 kB 00:00 2021-11-20T08:48:44.498 INFO:teuthology.orchestra.run.smithi093.stdout:(9/22): python3-chardet-3.0.4-7.el8.noarch.rpm 27 MB/s | 195 kB 00:00 2021-11-20T08:48:44.498 INFO:teuthology.orchestra.run.smithi093.stdout:(10/22): python3-idna-2.5-5.el8.noarch.rpm 21 MB/s | 97 kB 00:00 2021-11-20T08:48:44.499 INFO:teuthology.orchestra.run.smithi093.stdout:(11/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 12 MB/s | 34 kB 00:00 2021-11-20T08:48:44.500 INFO:teuthology.orchestra.run.smithi093.stdout:(12/22): python3-requests-2.20.0-2.1.el8_1.noar 19 MB/s | 123 kB 00:00 2021-11-20T08:48:44.500 INFO:teuthology.orchestra.run.smithi093.stdout:(13/22): python3-urllib3-1.24.2-5.el8.noarch.rp 21 MB/s | 177 kB 00:00 2021-11-20T08:48:44.500 INFO:teuthology.orchestra.run.smithi093.stdout:(14/22): conmon-2.0.30-1.module_el8.6.0+944+d41 13 MB/s | 54 kB 00:00 2021-11-20T08:48:44.536 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:48:44.539 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-11-20T08:48:44.539 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:48:44.539 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-11-20T08:48:44.539 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2021-11-20T08:48:44.540 INFO:teuthology.orchestra.run.smithi102.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T08:48:44.540 INFO:teuthology.orchestra.run.smithi102.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T08:48:44.540 INFO:teuthology.orchestra.run.smithi102.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T08:48:44.540 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T08:48:44.540 INFO:teuthology.orchestra.run.smithi102.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T08:48:44.541 INFO:teuthology.orchestra.run.smithi102.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T08:48:44.541 INFO:teuthology.orchestra.run.smithi102.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T08:48:44.541 INFO:teuthology.orchestra.run.smithi102.stdout:Installing group/module packages: 2021-11-20T08:48:44.541 INFO:teuthology.orchestra.run.smithi102.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T08:48:44.542 INFO:teuthology.orchestra.run.smithi102.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T08:48:44.542 INFO:teuthology.orchestra.run.smithi102.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T08:48:44.542 INFO:teuthology.orchestra.run.smithi102.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T08:48:44.542 INFO:teuthology.orchestra.run.smithi102.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T08:48:44.543 INFO:teuthology.orchestra.run.smithi102.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T08:48:44.543 INFO:teuthology.orchestra.run.smithi102.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T08:48:44.543 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:48:44.543 INFO:teuthology.orchestra.run.smithi102.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T08:48:44.543 INFO:teuthology.orchestra.run.smithi102.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T08:48:44.544 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T08:48:44.544 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T08:48:44.545 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T08:48:44.545 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T08:48:44.545 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T08:48:44.545 INFO:teuthology.orchestra.run.smithi102.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T08:48:44.545 INFO:teuthology.orchestra.run.smithi102.stdout:Installing module profiles: 2021-11-20T08:48:44.546 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools/common 2021-11-20T08:48:44.546 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling module streams: 2021-11-20T08:48:44.546 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools rhel8 2021-11-20T08:48:44.546 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:48:44.547 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:48:44.547 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-11-20T08:48:44.547 INFO:teuthology.orchestra.run.smithi102.stdout:Install 15 Packages 2021-11-20T08:48:44.547 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 7 Packages 2021-11-20T08:48:44.548 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:48:44.548 INFO:teuthology.orchestra.run.smithi093.stdout:(15/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 46 MB/s | 6.7 MB 00:00 2021-11-20T08:48:44.549 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 41 M 2021-11-20T08:48:44.549 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:48:44.555 INFO:teuthology.orchestra.run.smithi093.stdout:(16/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 37 MB/s | 518 kB 00:00 2021-11-20T08:48:44.560 INFO:teuthology.orchestra.run.smithi093.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 15 MB/s | 73 kB 00:00 2021-11-20T08:48:44.567 INFO:teuthology.orchestra.run.smithi093.stdout:(18/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 11 MB/s | 70 kB 00:00 2021-11-20T08:48:44.650 INFO:teuthology.orchestra.run.smithi102.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 1.8 MB/s | 193 kB 00:00 2021-11-20T08:48:44.651 INFO:teuthology.orchestra.run.smithi093.stdout:(19/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 37 MB/s | 3.1 MB 00:00 2021-11-20T08:48:44.661 INFO:teuthology.orchestra.run.smithi093.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 5.4 MB/s | 51 kB 00:00 2021-11-20T08:48:44.662 INFO:teuthology.orchestra.run.smithi102.stdout:(2/22): cockpit-podman-35-1.module_el8.6.0+944+ 3.8 MB/s | 457 kB 00:00 2021-11-20T08:48:44.663 INFO:teuthology.orchestra.run.smithi102.stdout:(3/22): python3-podman-3.2.1-4.module_el8.6.0+9 12 MB/s | 148 kB 00:00 2021-11-20T08:48:44.669 INFO:teuthology.orchestra.run.smithi102.stdout:(4/22): python3-pytoml-0.1.14-5.git7dea353.el8. 3.4 MB/s | 25 kB 00:00 2021-11-20T08:48:44.679 INFO:teuthology.orchestra.run.smithi102.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 5.7 MB/s | 94 kB 00:00 2021-11-20T08:48:44.714 INFO:teuthology.orchestra.run.smithi093.stdout:(21/22): buildah-1.23.1-2.module_el8.6.0+954+96 22 MB/s | 7.9 MB 00:00 2021-11-20T08:48:44.866 INFO:teuthology.orchestra.run.smithi102.stdout:(6/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 12 MB/s | 2.3 MB 00:00 2021-11-20T08:48:44.878 INFO:teuthology.orchestra.run.smithi102.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 3.9 MB/s | 48 kB 00:00 2021-11-20T08:48:44.892 INFO:teuthology.orchestra.run.smithi102.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 2.9 MB/s | 41 kB 00:00 2021-11-20T08:48:44.920 INFO:teuthology.orchestra.run.smithi102.stdout:(9/22): python3-chardet-3.0.4-7.el8.noarch.rpm 6.9 MB/s | 195 kB 00:00 2021-11-20T08:48:44.934 INFO:teuthology.orchestra.run.smithi102.stdout:(10/22): python3-idna-2.5-5.el8.noarch.rpm 7.2 MB/s | 97 kB 00:00 2021-11-20T08:48:44.941 INFO:teuthology.orchestra.run.smithi102.stdout:(11/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 4.6 MB/s | 34 kB 00:00 2021-11-20T08:48:44.955 INFO:teuthology.orchestra.run.smithi102.stdout:(12/22): python3-requests-2.20.0-2.1.el8_1.noar 9.1 MB/s | 123 kB 00:00 2021-11-20T08:48:44.985 INFO:teuthology.orchestra.run.smithi102.stdout:(13/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 21 MB/s | 6.7 MB 00:00 2021-11-20T08:48:44.986 INFO:teuthology.orchestra.run.smithi102.stdout:(14/22): python3-urllib3-1.24.2-5.el8.noarch.rp 5.6 MB/s | 177 kB 00:00 2021-11-20T08:48:44.987 INFO:teuthology.orchestra.run.smithi102.stdout:(15/22): conmon-2.0.30-1.module_el8.6.0+944+d41 28 MB/s | 54 kB 00:00 2021-11-20T08:48:45.040 INFO:teuthology.orchestra.run.smithi093.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 35 MB/s | 19 MB 00:00 2021-11-20T08:48:45.043 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:48:45.044 INFO:teuthology.orchestra.run.smithi093.stdout:Total 60 MB/s | 41 MB 00:00 2021-11-20T08:48:45.070 INFO:teuthology.orchestra.run.smithi102.stdout:(16/22): buildah-1.23.1-2.module_el8.6.0+954+96 15 MB/s | 7.9 MB 00:00 2021-11-20T08:48:45.072 INFO:teuthology.orchestra.run.smithi102.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 32 MB/s | 73 kB 00:00 2021-11-20T08:48:45.075 INFO:teuthology.orchestra.run.smithi102.stdout:(18/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 5.7 MB/s | 518 kB 00:00 2021-11-20T08:48:45.076 INFO:teuthology.orchestra.run.smithi102.stdout:(19/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 20 MB/s | 70 kB 00:00 2021-11-20T08:48:45.077 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:48:45.082 INFO:teuthology.orchestra.run.smithi102.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 8.4 MB/s | 51 kB 00:00 2021-11-20T08:48:45.105 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:48:45.105 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:48:45.119 INFO:teuthology.orchestra.run.smithi102.stdout:(21/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 70 MB/s | 3.1 MB 00:00 2021-11-20T08:48:45.397 INFO:teuthology.orchestra.run.smithi102.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 46 MB/s | 19 MB 00:00 2021-11-20T08:48:45.400 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:48:45.401 INFO:teuthology.orchestra.run.smithi102.stdout:Total 48 MB/s | 41 MB 00:00 2021-11-20T08:48:45.434 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:48:45.442 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:48:45.445 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:48:45.461 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:48:45.461 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:48:45.802 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:48:45.806 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:48:46.506 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:48:46.633 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T08:48:46.838 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T08:48:46.876 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:48:46.991 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T08:48:47.171 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T08:48:47.351 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T08:48:47.502 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T08:48:47.676 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T08:48:47.705 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T08:48:47.868 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T08:48:47.882 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T08:48:48.065 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T08:48:48.084 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T08:48:48.255 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T08:48:48.301 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T08:48:48.438 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T08:48:48.450 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T08:48:48.594 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T08:48:48.649 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T08:48:48.791 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T08:48:48.796 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T08:48:48.950 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T08:48:48.980 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T08:48:49.109 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T08:48:49.143 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T08:48:49.303 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T08:48:49.476 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T08:48:49.967 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T08:48:50.164 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T08:48:50.281 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T08:48:50.285 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T08:48:50.477 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T08:48:50.635 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T08:48:51.895 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T08:48:52.099 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T08:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T08:48:52.281 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T08:48:52.450 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T08:48:52.632 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T08:48:52.648 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T08:48:52.990 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T08:48:53.349 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T08:48:53.522 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T08:48:53.667 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T08:48:53.690 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T08:48:53.777 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T08:48:53.839 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T08:48:53.910 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T08:48:53.956 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T08:48:54.086 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T08:48:54.098 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T08:48:54.227 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T08:48:54.252 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T08:48:54.345 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T08:48:54.386 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T08:48:54.511 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T08:48:54.535 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T08:48:54.662 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T08:48:54.666 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T08:48:54.835 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T08:48:54.991 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T08:48:55.069 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T08:48:55.069 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T08:48:55.070 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T08:48:55.070 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T08:48:55.070 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T08:48:55.070 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T08:48:55.071 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T08:48:55.071 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T08:48:55.071 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T08:48:55.071 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T08:48:55.072 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T08:48:55.072 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T08:48:55.072 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T08:48:55.072 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T08:48:55.073 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T08:48:55.073 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T08:48:55.073 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T08:48:55.073 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T08:48:55.074 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T08:48:55.074 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T08:48:55.074 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T08:48:55.074 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T08:48:55.075 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T08:48:55.075 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T08:48:55.075 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T08:48:55.075 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T08:48:55.075 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T08:48:55.076 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T08:48:55.076 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T08:48:55.384 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T08:48:55.384 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T08:48:55.385 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T08:48:55.385 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T08:48:55.385 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T08:48:55.386 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T08:48:55.387 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T08:48:55.387 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T08:48:55.387 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T08:48:55.388 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T08:48:55.388 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T08:48:55.388 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T08:48:55.389 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T08:48:55.389 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T08:48:55.389 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T08:48:55.389 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T08:48:55.390 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T08:48:55.390 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T08:48:55.390 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T08:48:55.391 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T08:48:55.391 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T08:48:55.391 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T08:48:55.391 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T08:48:55.392 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T08:48:55.392 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T08:48:55.393 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T08:48:55.393 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T08:48:55.393 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T08:48:55.394 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T08:48:55.730 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T08:48:55.731 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:48:55.732 INFO:teuthology.orchestra.run.smithi093.stdout:Upgraded: 2021-11-20T08:48:55.732 INFO:teuthology.orchestra.run.smithi093.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T08:48:55.733 INFO:teuthology.orchestra.run.smithi093.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T08:48:55.733 INFO:teuthology.orchestra.run.smithi093.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:55.733 INFO:teuthology.orchestra.run.smithi093.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:55.734 INFO:teuthology.orchestra.run.smithi093.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:55.734 INFO:teuthology.orchestra.run.smithi093.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:55.735 INFO:teuthology.orchestra.run.smithi093.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:55.735 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:48:55.736 INFO:teuthology.orchestra.run.smithi093.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T08:48:55.736 INFO:teuthology.orchestra.run.smithi093.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T08:48:55.736 INFO:teuthology.orchestra.run.smithi093.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T08:48:55.737 INFO:teuthology.orchestra.run.smithi093.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T08:48:55.738 INFO:teuthology.orchestra.run.smithi093.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T08:48:55.738 INFO:teuthology.orchestra.run.smithi093.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T08:48:55.738 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T08:48:55.739 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T08:48:55.739 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T08:48:55.740 INFO:teuthology.orchestra.run.smithi093.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T08:48:55.740 INFO:teuthology.orchestra.run.smithi093.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T08:48:55.741 INFO:teuthology.orchestra.run.smithi093.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T08:48:55.741 INFO:teuthology.orchestra.run.smithi093.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T08:48:55.741 INFO:teuthology.orchestra.run.smithi093.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T08:48:55.742 INFO:teuthology.orchestra.run.smithi093.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T08:48:55.742 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:48:55.743 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:48:56.004 DEBUG:teuthology.parallel:result is None 2021-11-20T08:48:56.095 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T08:48:56.096 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:48:56.096 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2021-11-20T08:48:56.096 INFO:teuthology.orchestra.run.smithi102.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T08:48:56.097 INFO:teuthology.orchestra.run.smithi102.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T08:48:56.097 INFO:teuthology.orchestra.run.smithi102.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:56.097 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:56.097 INFO:teuthology.orchestra.run.smithi102.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:56.098 INFO:teuthology.orchestra.run.smithi102.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:56.098 INFO:teuthology.orchestra.run.smithi102.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T08:48:56.098 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:48:56.098 INFO:teuthology.orchestra.run.smithi102.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T08:48:56.099 INFO:teuthology.orchestra.run.smithi102.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T08:48:56.099 INFO:teuthology.orchestra.run.smithi102.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T08:48:56.099 INFO:teuthology.orchestra.run.smithi102.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T08:48:56.099 INFO:teuthology.orchestra.run.smithi102.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T08:48:56.100 INFO:teuthology.orchestra.run.smithi102.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T08:48:56.100 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T08:48:56.100 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T08:48:56.100 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T08:48:56.100 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T08:48:56.101 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T08:48:56.101 INFO:teuthology.orchestra.run.smithi102.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T08:48:56.101 INFO:teuthology.orchestra.run.smithi102.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T08:48:56.101 INFO:teuthology.orchestra.run.smithi102.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T08:48:56.102 INFO:teuthology.orchestra.run.smithi102.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T08:48:56.102 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:48:56.102 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:48:56.373 DEBUG:teuthology.parallel:result is None 2021-11-20T08:48:56.374 INFO:teuthology.run_tasks:Running task install... 2021-11-20T08:48:56.387 DEBUG:teuthology.task.install:project ceph 2021-11-20T08:48:56.388 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'}} 2021-11-20T08:48:56.388 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T08:48:56.389 INFO:teuthology.task.install:Using flavor: default 2021-11-20T08:48:56.396 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-11-20T08:48:56.396 INFO:teuthology.task.install:extra packages: [] 2021-11-20T08:48:56.397 DEBUG:teuthology.orchestra.run.smithi093:> sudo lsb_release -is 2021-11-20T08:48:56.432 INFO:teuthology.orchestra.run.smithi093.stdout:CentOSStream 2021-11-20T08:48:56.434 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T08:48:56.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo lsb_release -is 2021-11-20T08:48:56.435 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T08:48:56.436 DEBUG:teuthology.orchestra.run.smithi093:> sudo lsb_release -is 2021-11-20T08:48:56.471 INFO:teuthology.orchestra.run.smithi102.stdout:CentOSStream 2021-11-20T08:48:56.473 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T08:48:56.474 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T08:48:56.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo lsb_release -is 2021-11-20T08:48:56.507 INFO:teuthology.orchestra.run.smithi093.stdout:CentOSStream 2021-11-20T08:48:56.509 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T08:48:56.509 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T08:48:56.509 INFO:teuthology.packaging:ref: None 2021-11-20T08:48:56.510 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T08:48:56.510 INFO:teuthology.packaging:branch: None 2021-11-20T08:48:56.510 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T08:48:56.746 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:48:56.747 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:48:56.787 INFO:teuthology.orchestra.run.smithi102.stdout:CentOSStream 2021-11-20T08:48:56.797 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T08:48:56.798 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T08:48:56.798 INFO:teuthology.packaging:ref: None 2021-11-20T08:48:56.799 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T08:48:56.799 INFO:teuthology.packaging:branch: None 2021-11-20T08:48:56.799 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T08:48:57.022 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:48:57.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:48:57.168 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T08:48:57.169 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T08:48:57.203 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T08:48:57.203 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:48:57.204 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T08:48:57.232 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T08:48:57.232 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T08:48:57.233 INFO:teuthology.packaging:ref: None 2021-11-20T08:48:57.233 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T08:48:57.233 INFO:teuthology.packaging:branch: None 2021-11-20T08:48:57.234 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T08:48:57.234 DEBUG:teuthology.orchestra.run.smithi093:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T08:48:57.237 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T08:48:57.237 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T08:48:57.265 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T08:48:57.265 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:48:57.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T08:48:57.291 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T08:48:57.292 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T08:48:57.292 INFO:teuthology.packaging:ref: None 2021-11-20T08:48:57.292 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T08:48:57.293 INFO:teuthology.packaging:branch: None 2021-11-20T08:48:57.293 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T08:48:57.293 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T08:48:57.299 DEBUG:teuthology.orchestra.run.smithi093:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T08:48:57.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T08:48:57.374 DEBUG:teuthology.orchestra.run.smithi093:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T08:48:57.433 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T08:48:57.439 INFO:teuthology.orchestra.run.smithi093.stdout:check_obsoletes = 1 2021-11-20T08:48:57.440 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum clean all 2021-11-20T08:48:57.497 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2021-11-20T08:48:57.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2021-11-20T08:48:57.675 INFO:teuthology.orchestra.run.smithi093.stdout:75 files removed 2021-11-20T08:48:57.695 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-radosgw 2021-11-20T08:48:57.732 INFO:teuthology.orchestra.run.smithi102.stdout:75 files removed 2021-11-20T08:48:57.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2021-11-20T08:48:58.057 INFO:teuthology.orchestra.run.smithi093.stdout:ceph packages for x86_64 2.8 MB/s | 74 kB 00:00 2021-11-20T08:48:58.113 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 2.8 MB/s | 74 kB 00:00 2021-11-20T08:48:58.117 INFO:teuthology.orchestra.run.smithi093.stdout:ceph noarch packages 642 kB/s | 15 kB 00:00 2021-11-20T08:48:58.169 INFO:teuthology.orchestra.run.smithi093.stdout:ceph source packages 71 kB/s | 1.7 kB 00:00 2021-11-20T08:48:58.173 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 634 kB/s | 15 kB 00:00 2021-11-20T08:48:58.225 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 72 kB/s | 1.7 kB 00:00 2021-11-20T08:48:58.489 INFO:teuthology.orchestra.run.smithi093.stdout:CentOS-8 - AppStream 58 MB/s | 17 MB 00:00 2021-11-20T08:48:58.616 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - AppStream 47 MB/s | 17 MB 00:00 2021-11-20T08:49:01.595 INFO:teuthology.orchestra.run.smithi093.stdout:CentOS-8 - Base 61 MB/s | 15 MB 00:00 2021-11-20T08:49:01.777 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Base 55 MB/s | 15 MB 00:00 2021-11-20T08:49:03.778 INFO:teuthology.orchestra.run.smithi093.stdout:CentOS-8 - Extras 220 kB/s | 16 kB 00:00 2021-11-20T08:49:04.052 INFO:teuthology.orchestra.run.smithi093.stdout:CentOS-8 - PowerTools 21 MB/s | 3.6 MB 00:00 2021-11-20T08:49:04.072 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - Extras 201 kB/s | 16 kB 00:00 2021-11-20T08:49:04.286 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-8 - PowerTools 32 MB/s | 3.6 MB 00:00 2021-11-20T08:49:06.609 INFO:teuthology.orchestra.run.smithi093.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2021-11-20T08:49:07.278 INFO:teuthology.orchestra.run.smithi102.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-11-20T08:49:07.989 INFO:teuthology.orchestra.run.smithi093.stdout:Apache Arrow 144 kB/s | 184 kB 00:01 2021-11-20T08:49:08.481 INFO:teuthology.orchestra.run.smithi102.stdout:Apache Arrow 167 kB/s | 184 kB 00:01 2021-11-20T08:49:09.836 INFO:teuthology.orchestra.run.smithi093.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 11 MB 00:01 2021-11-20T08:49:10.612 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 11 MB 00:01 2021-11-20T08:49:12.207 INFO:teuthology.orchestra.run.smithi093.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-11-20T08:49:12.989 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-11-20T08:49:14.927 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:49:14.930 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:49:14.930 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:49:14.930 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:49:14.930 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:49:14.931 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T08:49:14.931 INFO:teuthology.orchestra.run.smithi093.stdout:Upgrading: 2021-11-20T08:49:14.931 INFO:teuthology.orchestra.run.smithi093.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T08:49:14.931 INFO:teuthology.orchestra.run.smithi093.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T08:49:14.932 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:49:14.932 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T08:49:14.932 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T08:49:14.932 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T08:49:14.933 INFO:teuthology.orchestra.run.smithi093.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T08:49:14.933 INFO:teuthology.orchestra.run.smithi093.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T08:49:14.933 INFO:teuthology.orchestra.run.smithi093.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T08:49:14.933 INFO:teuthology.orchestra.run.smithi093.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T08:49:14.934 INFO:teuthology.orchestra.run.smithi093.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T08:49:14.934 INFO:teuthology.orchestra.run.smithi093.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T08:49:14.934 INFO:teuthology.orchestra.run.smithi093.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T08:49:14.934 INFO:teuthology.orchestra.run.smithi093.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T08:49:14.934 INFO:teuthology.orchestra.run.smithi093.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T08:49:14.935 INFO:teuthology.orchestra.run.smithi093.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T08:49:14.935 INFO:teuthology.orchestra.run.smithi093.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T08:49:14.935 INFO:teuthology.orchestra.run.smithi093.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T08:49:14.936 INFO:teuthology.orchestra.run.smithi093.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T08:49:14.936 INFO:teuthology.orchestra.run.smithi093.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T08:49:14.936 INFO:teuthology.orchestra.run.smithi093.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T08:49:14.936 INFO:teuthology.orchestra.run.smithi093.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T08:49:14.936 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T08:49:14.937 INFO:teuthology.orchestra.run.smithi093.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T08:49:14.937 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T08:49:14.937 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T08:49:14.937 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T08:49:14.937 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:49:14.938 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:49:14.938 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:49:14.939 INFO:teuthology.orchestra.run.smithi093.stdout:Install 25 Packages 2021-11-20T08:49:14.939 INFO:teuthology.orchestra.run.smithi093.stdout:Upgrade 2 Packages 2021-11-20T08:49:14.940 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:49:14.940 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 59 M 2021-11-20T08:49:14.941 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:49:15.156 INFO:teuthology.orchestra.run.smithi093.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 27 MB/s | 6.0 MB 00:00 2021-11-20T08:49:15.158 INFO:teuthology.orchestra.run.smithi093.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 11 MB/s | 25 kB 00:00 2021-11-20T08:49:15.193 INFO:teuthology.orchestra.run.smithi093.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2021-11-20T08:49:15.307 INFO:teuthology.orchestra.run.smithi093.stdout:(4/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 34 MB/s | 13 MB 00:00 2021-11-20T08:49:15.418 INFO:teuthology.orchestra.run.smithi093.stdout:(5/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 4.1 MB/s | 486 kB 00:00 2021-11-20T08:49:15.419 INFO:teuthology.orchestra.run.smithi093.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 7.4 MB/s | 44 kB 00:00 2021-11-20T08:49:15.419 INFO:teuthology.orchestra.run.smithi093.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 34 MB/s | 80 kB 00:00 2021-11-20T08:49:15.420 INFO:teuthology.orchestra.run.smithi093.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 26 MB/s | 213 kB 00:00 2021-11-20T08:49:15.420 INFO:teuthology.orchestra.run.smithi093.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 24 MB/s | 387 kB 00:00 2021-11-20T08:49:15.420 INFO:teuthology.orchestra.run.smithi093.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 27 MB/s | 368 kB 00:00 2021-11-20T08:49:15.421 INFO:teuthology.orchestra.run.smithi093.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 37 MB/s | 114 kB 00:00 2021-11-20T08:49:15.421 INFO:teuthology.orchestra.run.smithi093.stdout:(12/27): librgw2-16.2.4-0.el8.x86_64.rpm 54 MB/s | 4.1 MB 00:00 2021-11-20T08:49:15.421 INFO:teuthology.orchestra.run.smithi093.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 4.2 MB/s | 145 kB 00:00 2021-11-20T08:49:15.421 INFO:teuthology.orchestra.run.smithi093.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 56 MB/s | 259 kB 00:00 2021-11-20T08:49:15.422 INFO:teuthology.orchestra.run.smithi093.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 31 MB/s | 44 kB 00:00 2021-11-20T08:49:15.422 INFO:teuthology.orchestra.run.smithi093.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 27 MB/s | 41 kB 00:00 2021-11-20T08:49:15.423 INFO:teuthology.orchestra.run.smithi093.stdout:(17/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 36 MB/s | 47 kB 00:00 2021-11-20T08:49:15.423 INFO:teuthology.orchestra.run.smithi093.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 42 MB/s | 78 kB 00:00 2021-11-20T08:49:15.423 INFO:teuthology.orchestra.run.smithi093.stdout:(19/27): librdkafka-0.11.4-3.el8.x86_64.rpm 10 MB/s | 354 kB 00:00 2021-11-20T08:49:15.685 INFO:teuthology.orchestra.run.smithi093.stdout:(20/27): ceph-common-16.2.4-0.el8.x86_64.rpm 32 MB/s | 24 MB 00:00 2021-11-20T08:49:15.685 INFO:teuthology.orchestra.run.smithi093.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 417 kB/s | 116 kB 00:00 2021-11-20T08:49:15.716 INFO:teuthology.orchestra.run.smithi093.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2021-11-20T08:49:15.716 INFO:teuthology.orchestra.run.smithi093.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.9 MB/s | 59 kB 00:00 2021-11-20T08:49:15.745 INFO:teuthology.orchestra.run.smithi093.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2021-11-20T08:49:15.747 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:49:15.750 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:49:15.750 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:49:15.751 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:49:15.751 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:49:15.751 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T08:49:15.751 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2021-11-20T08:49:15.752 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T08:49:15.752 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T08:49:15.752 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:49:15.752 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T08:49:15.753 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T08:49:15.753 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T08:49:15.753 INFO:teuthology.orchestra.run.smithi102.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T08:49:15.753 INFO:teuthology.orchestra.run.smithi102.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T08:49:15.754 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T08:49:15.754 INFO:teuthology.orchestra.run.smithi102.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T08:49:15.754 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T08:49:15.754 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T08:49:15.754 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T08:49:15.755 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T08:49:15.755 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T08:49:15.755 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T08:49:15.756 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T08:49:15.756 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T08:49:15.757 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T08:49:15.757 INFO:teuthology.orchestra.run.smithi102.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T08:49:15.758 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T08:49:15.758 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T08:49:15.758 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T08:49:15.759 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T08:49:15.759 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T08:49:15.759 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T08:49:15.759 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T08:49:15.760 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:49:15.760 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:49:15.760 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:49:15.760 INFO:teuthology.orchestra.run.smithi102.stdout:Install 25 Packages 2021-11-20T08:49:15.761 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2021-11-20T08:49:15.761 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:49:15.761 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 59 M 2021-11-20T08:49:15.762 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:49:15.763 INFO:teuthology.orchestra.run.smithi093.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 80 MB/s | 3.7 MB 00:00 2021-11-20T08:49:15.821 INFO:teuthology.orchestra.run.smithi093.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 53 MB/s | 4.0 MB 00:00 2021-11-20T08:49:15.823 INFO:teuthology.orchestra.run.smithi093.stdout:(27/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-11-20T08:49:15.827 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:49:15.828 INFO:teuthology.orchestra.run.smithi093.stdout:Total 65 MB/s | 59 MB 00:00 2021-11-20T08:49:15.839 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:49:15.886 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:49:15.886 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:49:15.987 INFO:teuthology.orchestra.run.smithi102.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 26 MB/s | 6.0 MB 00:00 2021-11-20T08:49:15.989 INFO:teuthology.orchestra.run.smithi102.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 14 MB/s | 25 kB 00:00 2021-11-20T08:49:16.011 INFO:teuthology.orchestra.run.smithi102.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 801 kB 00:00 2021-11-20T08:49:16.026 INFO:teuthology.orchestra.run.smithi102.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 32 MB/s | 486 kB 00:00 2021-11-20T08:49:16.144 INFO:teuthology.orchestra.run.smithi102.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.1 MB 00:00 2021-11-20T08:49:16.146 INFO:teuthology.orchestra.run.smithi102.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 29 MB/s | 44 kB 00:00 2021-11-20T08:49:16.148 INFO:teuthology.orchestra.run.smithi102.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 36 MB/s | 80 kB 00:00 2021-11-20T08:49:16.155 INFO:teuthology.orchestra.run.smithi102.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 32 MB/s | 213 kB 00:00 2021-11-20T08:49:16.167 INFO:teuthology.orchestra.run.smithi102.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 33 MB/s | 387 kB 00:00 2021-11-20T08:49:16.179 INFO:teuthology.orchestra.run.smithi102.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 32 MB/s | 368 kB 00:00 2021-11-20T08:49:16.181 INFO:teuthology.orchestra.run.smithi102.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 39 MB/s | 114 kB 00:00 2021-11-20T08:49:16.214 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:49:16.217 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:49:16.291 INFO:teuthology.orchestra.run.smithi102.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 24 MB/s | 13 MB 00:00 2021-11-20T08:49:16.308 INFO:teuthology.orchestra.run.smithi102.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 1.1 MB/s | 145 kB 00:00 2021-11-20T08:49:16.313 INFO:teuthology.orchestra.run.smithi102.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 48 MB/s | 259 kB 00:00 2021-11-20T08:49:16.314 INFO:teuthology.orchestra.run.smithi102.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 32 MB/s | 44 kB 00:00 2021-11-20T08:49:16.316 INFO:teuthology.orchestra.run.smithi102.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 29 MB/s | 41 kB 00:00 2021-11-20T08:49:16.318 INFO:teuthology.orchestra.run.smithi102.stdout:(17/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 34 MB/s | 47 kB 00:00 2021-11-20T08:49:16.320 INFO:teuthology.orchestra.run.smithi102.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 40 MB/s | 78 kB 00:00 2021-11-20T08:49:16.332 INFO:teuthology.orchestra.run.smithi102.stdout:(19/27): librdkafka-0.11.4-3.el8.x86_64.rpm 8.7 MB/s | 354 kB 00:00 2021-11-20T08:49:16.501 INFO:teuthology.orchestra.run.smithi102.stdout:(20/27): ceph-common-16.2.4-0.el8.x86_64.rpm 32 MB/s | 24 MB 00:00 2021-11-20T08:49:16.566 INFO:teuthology.orchestra.run.smithi102.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 473 kB/s | 116 kB 00:00 2021-11-20T08:49:16.568 INFO:teuthology.orchestra.run.smithi102.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2021-11-20T08:49:16.598 INFO:teuthology.orchestra.run.smithi102.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2021-11-20T08:49:16.598 INFO:teuthology.orchestra.run.smithi102.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2021-11-20T08:49:16.671 INFO:teuthology.orchestra.run.smithi102.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2021-11-20T08:49:16.686 INFO:teuthology.orchestra.run.smithi102.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 46 MB/s | 4.0 MB 00:00 2021-11-20T08:49:16.687 INFO:teuthology.orchestra.run.smithi102.stdout:(27/27): leveldb-1.22-1.el8.x86_64.rpm 971 kB/s | 181 kB 00:00 2021-11-20T08:49:16.691 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:49:16.692 INFO:teuthology.orchestra.run.smithi102.stdout:Total 62 MB/s | 59 MB 00:00 2021-11-20T08:49:16.703 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:49:16.751 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:49:16.751 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:49:17.083 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:49:17.086 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:49:17.316 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:49:17.454 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T08:49:17.641 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T08:49:17.773 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T08:49:18.186 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:49:18.186 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T08:49:18.329 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T08:49:18.339 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T08:49:18.425 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T08:49:18.499 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T08:49:18.639 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T08:49:18.657 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T08:49:18.765 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T08:49:18.829 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T08:49:18.987 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T08:49:19.067 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T08:49:19.147 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T08:49:19.190 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T08:49:19.290 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T08:49:19.480 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T08:49:19.549 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T08:49:19.607 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T08:49:19.652 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T08:49:19.664 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T08:49:19.744 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T08:49:19.804 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T08:49:19.881 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T08:49:19.973 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T08:49:20.022 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T08:49:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T08:49:20.211 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T08:49:20.340 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T08:49:20.374 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T08:49:20.481 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T08:49:20.507 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T08:49:20.560 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T08:49:20.584 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T08:49:20.697 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T08:49:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T08:49:20.848 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T08:49:20.917 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T08:49:21.036 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T08:49:21.122 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T08:49:21.182 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T08:49:21.290 INFO:teuthology.orchestra.run.smithi093.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T08:49:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T08:49:21.366 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T08:49:21.383 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T08:49:21.499 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T08:49:21.559 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T08:49:21.724 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T08:49:21.822 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T08:49:21.892 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T08:49:21.926 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T08:49:22.063 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T08:49:22.064 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T08:49:22.148 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T08:49:22.214 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T08:49:22.326 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T08:49:22.401 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T08:49:22.524 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T08:49:22.579 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T08:49:22.648 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T08:49:22.857 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T08:49:23.599 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T08:49:23.817 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T08:49:24.021 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T08:49:24.215 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T08:49:27.458 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T08:49:27.656 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T08:49:28.355 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T08:49:28.491 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T08:49:28.593 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T08:49:28.672 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T08:49:28.694 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T08:49:28.988 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T08:49:29.368 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T08:49:29.640 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T08:49:29.746 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T08:49:29.863 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T08:49:46.732 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T08:49:46.733 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /sys 2021-11-20T08:49:46.733 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /proc 2021-11-20T08:49:46.734 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /dev 2021-11-20T08:49:46.734 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /run 2021-11-20T08:49:46.734 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /mnt 2021-11-20T08:49:46.735 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /var/tmp 2021-11-20T08:49:46.735 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /home 2021-11-20T08:49:46.735 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /tmp 2021-11-20T08:49:46.736 INFO:teuthology.orchestra.run.smithi093.stdout:skipping the directory /dev 2021-11-20T08:49:46.736 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:49:48.014 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T08:49:48.090 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T08:49:48.091 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T08:49:48.233 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T08:49:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2021-11-20T08:49:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2021-11-20T08:49:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2021-11-20T08:49:48.234 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /run 2021-11-20T08:49:48.235 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2021-11-20T08:49:48.235 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2021-11-20T08:49:48.235 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2021-11-20T08:49:48.235 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2021-11-20T08:49:48.236 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2021-11-20T08:49:48.236 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:49:48.271 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T08:49:48.271 INFO:teuthology.orchestra.run.smithi093.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T08:49:49.016 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T08:49:49.017 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T08:49:49.017 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T08:49:49.017 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T08:49:49.018 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T08:49:49.018 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T08:49:49.018 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T08:49:49.018 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T08:49:49.019 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T08:49:49.019 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T08:49:49.019 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T08:49:49.019 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T08:49:49.020 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T08:49:49.020 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T08:49:49.020 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T08:49:49.020 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T08:49:49.020 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T08:49:49.021 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T08:49:49.021 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T08:49:49.021 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T08:49:49.021 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T08:49:49.022 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T08:49:49.022 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T08:49:49.022 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T08:49:49.022 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T08:49:49.023 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T08:49:49.023 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T08:49:49.023 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T08:49:49.023 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T08:49:49.520 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T08:49:49.582 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T08:49:49.582 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T08:49:49.771 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T08:49:49.772 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T08:49:49.907 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T08:49:49.908 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:49:49.908 INFO:teuthology.orchestra.run.smithi093.stdout:Upgraded: 2021-11-20T08:49:49.908 INFO:teuthology.orchestra.run.smithi093.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.908 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:49:49.908 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.909 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.909 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.909 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.909 INFO:teuthology.orchestra.run.smithi093.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T08:49:49.910 INFO:teuthology.orchestra.run.smithi093.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T08:49:49.910 INFO:teuthology.orchestra.run.smithi093.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T08:49:49.910 INFO:teuthology.orchestra.run.smithi093.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T08:49:49.910 INFO:teuthology.orchestra.run.smithi093.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.911 INFO:teuthology.orchestra.run.smithi093.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T08:49:49.911 INFO:teuthology.orchestra.run.smithi093.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T08:49:49.911 INFO:teuthology.orchestra.run.smithi093.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T08:49:49.911 INFO:teuthology.orchestra.run.smithi093.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.911 INFO:teuthology.orchestra.run.smithi093.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T08:49:49.912 INFO:teuthology.orchestra.run.smithi093.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.912 INFO:teuthology.orchestra.run.smithi093.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T08:49:49.912 INFO:teuthology.orchestra.run.smithi093.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T08:49:49.912 INFO:teuthology.orchestra.run.smithi093.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T08:49:49.912 INFO:teuthology.orchestra.run.smithi093.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.913 INFO:teuthology.orchestra.run.smithi093.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.913 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.913 INFO:teuthology.orchestra.run.smithi093.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T08:49:49.913 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.914 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.914 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:49.914 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:49:49.914 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:49:50.218 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-test 2021-11-20T08:49:50.441 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T08:49:50.442 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T08:49:50.443 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T08:49:50.443 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T08:49:50.443 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T08:49:50.444 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T08:49:50.444 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T08:49:50.445 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T08:49:50.445 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T08:49:50.445 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T08:49:50.445 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T08:49:50.446 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T08:49:50.446 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T08:49:50.446 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T08:49:50.447 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T08:49:50.447 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T08:49:50.447 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T08:49:50.448 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T08:49:50.448 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T08:49:50.448 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T08:49:50.448 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T08:49:50.449 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T08:49:50.449 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T08:49:50.449 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T08:49:50.450 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T08:49:50.450 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T08:49:50.450 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T08:49:50.450 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T08:49:50.451 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T08:49:50.768 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:00:38 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:49:51.382 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T08:49:51.383 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:49:51.383 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2021-11-20T08:49:51.383 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.384 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:49:51.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.385 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.385 INFO:teuthology.orchestra.run.smithi102.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T08:49:51.385 INFO:teuthology.orchestra.run.smithi102.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T08:49:51.385 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T08:49:51.386 INFO:teuthology.orchestra.run.smithi102.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T08:49:51.386 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.386 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T08:49:51.386 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T08:49:51.387 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T08:49:51.387 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.387 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T08:49:51.387 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.387 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T08:49:51.388 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T08:49:51.389 INFO:teuthology.orchestra.run.smithi102.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T08:49:51.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T08:49:51.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T08:49:51.392 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:49:51.392 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:49:51.659 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2021-11-20T08:49:52.068 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:49:52.068 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:49:52.069 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:49:52.069 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:49:52.069 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:49:52.069 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T08:49:52.069 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:49:52.070 INFO:teuthology.orchestra.run.smithi093.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T08:49:52.070 INFO:teuthology.orchestra.run.smithi093.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T08:49:52.070 INFO:teuthology.orchestra.run.smithi093.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T08:49:52.070 INFO:teuthology.orchestra.run.smithi093.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T08:49:52.071 INFO:teuthology.orchestra.run.smithi093.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T08:49:52.071 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:49:52.071 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:49:52.071 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:49:52.072 INFO:teuthology.orchestra.run.smithi093.stdout:Install 6 Packages 2021-11-20T08:49:52.072 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:49:52.073 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 53 M 2021-11-20T08:49:52.073 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 229 M 2021-11-20T08:49:52.073 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:49:52.098 INFO:teuthology.orchestra.run.smithi093.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 6.4 MB/s | 169 kB 00:00 2021-11-20T08:49:52.117 INFO:teuthology.orchestra.run.smithi093.stdout:(2/6): jq-1.6-3.el8.x86_64.rpm 4.3 MB/s | 202 kB 00:00 2021-11-20T08:49:52.122 INFO:teuthology.orchestra.run.smithi093.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 64 MB/s | 323 kB 00:00 2021-11-20T08:49:52.130 INFO:teuthology.orchestra.run.smithi093.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.7 MB/s | 187 kB 00:00 2021-11-20T08:49:52.208 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:40 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:49:52.381 INFO:teuthology.orchestra.run.smithi093.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 267 kB/s | 69 kB 00:00 2021-11-20T08:49:52.971 INFO:teuthology.orchestra.run.smithi093.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 58 MB/s | 52 MB 00:00 2021-11-20T08:49:52.975 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:49:52.975 INFO:teuthology.orchestra.run.smithi093.stdout:Total 58 MB/s | 53 MB 00:00 2021-11-20T08:49:52.981 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:49:53.005 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:49:53.006 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:49:53.289 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:49:53.292 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:49:53.529 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:49:53.531 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:49:53.531 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:49:53.531 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:49:53.532 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:49:53.532 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T08:49:53.532 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:49:53.532 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T08:49:53.533 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T08:49:53.533 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T08:49:53.533 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T08:49:53.533 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T08:49:53.533 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:49:53.534 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:49:53.534 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:49:53.534 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2021-11-20T08:49:53.534 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:49:53.535 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 53 M 2021-11-20T08:49:53.536 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 229 M 2021-11-20T08:49:53.536 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:49:53.558 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 6.6 MB/s | 169 kB 00:00 2021-11-20T08:49:53.819 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:49:53.980 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T08:49:54.179 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T08:49:54.326 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T08:49:54.395 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 60 MB/s | 52 MB 00:00 2021-11-20T08:49:54.404 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T08:49:54.541 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T08:49:54.693 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T08:49:55.211 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): jq-1.6-3.el8.x86_64.rpm 120 kB/s | 202 kB 00:01 2021-11-20T08:49:55.214 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 113 kB/s | 187 kB 00:01 2021-11-20T08:49:55.238 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 383 kB/s | 323 kB 00:00 2021-11-20T08:49:55.345 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 516 kB/s | 69 kB 00:00 2021-11-20T08:49:55.350 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:49:55.351 INFO:teuthology.orchestra.run.smithi102.stdout:Total 29 MB/s | 53 MB 00:01 2021-11-20T08:49:55.358 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:49:55.381 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:49:55.382 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:49:55.667 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:49:55.669 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:49:56.188 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:49:56.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T08:49:56.547 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T08:49:56.710 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T08:49:56.788 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T08:49:56.942 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T08:49:57.126 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T08:49:59.483 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T08:49:59.707 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T08:49:59.959 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T08:49:59.960 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T08:49:59.961 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T08:49:59.961 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T08:49:59.961 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T08:49:59.963 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T08:50:00.350 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T08:50:00.350 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:00.351 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:00.351 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T08:50:00.351 INFO:teuthology.orchestra.run.smithi093.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T08:50:00.351 INFO:teuthology.orchestra.run.smithi093.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T08:50:00.352 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:00.352 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:00.537 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph 2021-11-20T08:50:01.078 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:00:49 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:01.911 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T08:50:02.149 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T08:50:02.353 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:02.356 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:02.357 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:50:02.357 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:02.357 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:50:02.358 INFO:teuthology.orchestra.run.smithi093.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T08:50:02.358 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:50:02.359 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T08:50:02.359 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T08:50:02.360 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T08:50:02.360 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T08:50:02.361 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T08:50:02.361 INFO:teuthology.orchestra.run.smithi093.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T08:50:02.362 INFO:teuthology.orchestra.run.smithi093.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T08:50:02.362 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T08:50:02.362 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T08:50:02.363 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T08:50:02.363 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T08:50:02.364 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T08:50:02.364 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T08:50:02.364 INFO:teuthology.orchestra.run.smithi093.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T08:50:02.365 INFO:teuthology.orchestra.run.smithi093.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T08:50:02.365 INFO:teuthology.orchestra.run.smithi093.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T08:50:02.365 INFO:teuthology.orchestra.run.smithi093.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T08:50:02.365 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T08:50:02.366 INFO:teuthology.orchestra.run.smithi093.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T08:50:02.366 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T08:50:02.366 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T08:50:02.366 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T08:50:02.367 INFO:teuthology.orchestra.run.smithi093.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T08:50:02.367 INFO:teuthology.orchestra.run.smithi093.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T08:50:02.368 INFO:teuthology.orchestra.run.smithi093.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T08:50:02.368 INFO:teuthology.orchestra.run.smithi093.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T08:50:02.369 INFO:teuthology.orchestra.run.smithi093.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T08:50:02.369 INFO:teuthology.orchestra.run.smithi093.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T08:50:02.369 INFO:teuthology.orchestra.run.smithi093.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T08:50:02.370 INFO:teuthology.orchestra.run.smithi093.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T08:50:02.370 INFO:teuthology.orchestra.run.smithi093.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T08:50:02.371 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:02.371 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:50:02.372 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:02.372 INFO:teuthology.orchestra.run.smithi093.stdout:Install 32 Packages 2021-11-20T08:50:02.373 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:02.374 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 31 M 2021-11-20T08:50:02.375 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 111 M 2021-11-20T08:50:02.375 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:50:02.390 INFO:teuthology.orchestra.run.smithi093.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 225 kB/s | 6.3 kB 00:00 2021-11-20T08:50:02.404 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T08:50:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T08:50:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T08:50:02.405 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T08:50:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T08:50:02.406 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T08:50:02.440 INFO:teuthology.orchestra.run.smithi093.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 20 MB/s | 1.6 MB 00:00 2021-11-20T08:50:02.453 INFO:teuthology.orchestra.run.smithi093.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2021-11-20T08:50:02.462 INFO:teuthology.orchestra.run.smithi093.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 25 MB/s | 220 kB 00:00 2021-11-20T08:50:02.498 INFO:teuthology.orchestra.run.smithi093.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 4.3 MB/s | 157 kB 00:00 2021-11-20T08:50:02.500 INFO:teuthology.orchestra.run.smithi093.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2021-11-20T08:50:02.503 INFO:teuthology.orchestra.run.smithi093.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 37 MB/s | 103 kB 00:00 2021-11-20T08:50:02.506 INFO:teuthology.orchestra.run.smithi093.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 26 MB/s | 54 kB 00:00 2021-11-20T08:50:02.512 INFO:teuthology.orchestra.run.smithi093.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 66 MB/s | 457 kB 00:00 2021-11-20T08:50:02.516 INFO:teuthology.orchestra.run.smithi093.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 64 MB/s | 237 kB 00:00 2021-11-20T08:50:02.523 INFO:teuthology.orchestra.run.smithi093.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 76 MB/s | 559 kB 00:00 2021-11-20T08:50:02.526 INFO:teuthology.orchestra.run.smithi093.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 53 MB/s | 109 kB 00:00 2021-11-20T08:50:02.620 INFO:teuthology.orchestra.run.smithi093.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2021-11-20T08:50:02.640 INFO:teuthology.orchestra.run.smithi093.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 382 kB/s | 44 kB 00:00 2021-11-20T08:50:02.728 INFO:teuthology.orchestra.run.smithi093.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.9 MB/s | 173 kB 00:00 2021-11-20T08:50:02.878 INFO:teuthology.orchestra.run.smithi093.stdout:(16/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 42 MB/s | 18 MB 00:00 2021-11-20T08:50:02.879 INFO:teuthology.orchestra.run.smithi093.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 715 kB/s | 185 kB 00:00 2021-11-20T08:50:02.881 INFO:teuthology.orchestra.run.smithi093.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2021-11-20T08:50:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T08:50:02.890 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:02.890 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:50:02.890 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T08:50:02.890 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T08:50:02.891 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T08:50:02.891 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:02.891 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:02.909 INFO:teuthology.orchestra.run.smithi093.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 589 kB/s | 18 kB 00:00 2021-11-20T08:50:02.911 INFO:teuthology.orchestra.run.smithi093.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.6 MB/s | 49 kB 00:00 2021-11-20T08:50:02.936 INFO:teuthology.orchestra.run.smithi093.stdout:(21/32): python3-jaraco-6.2-6.el8.noarch.rpm 192 kB/s | 11 kB 00:00 2021-11-20T08:50:02.938 INFO:teuthology.orchestra.run.smithi093.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T08:50:02.971 INFO:teuthology.orchestra.run.smithi093.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 447 kB/s | 16 kB 00:00 2021-11-20T08:50:02.973 INFO:teuthology.orchestra.run.smithi093.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 526 kB/s | 19 kB 00:00 2021-11-20T08:50:03.015 INFO:teuthology.orchestra.run.smithi093.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 556 kB/s | 24 kB 00:00 2021-11-20T08:50:03.015 INFO:teuthology.orchestra.run.smithi093.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 710 kB/s | 29 kB 00:00 2021-11-20T08:50:03.016 INFO:teuthology.orchestra.run.smithi093.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.6 MB/s | 283 kB 00:00 2021-11-20T08:50:03.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2021-11-20T08:50:03.043 INFO:teuthology.orchestra.run.smithi093.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 975 kB/s | 27 kB 00:00 2021-11-20T08:50:03.071 INFO:teuthology.orchestra.run.smithi093.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.2 MB/s | 241 kB 00:00 2021-11-20T08:50:03.099 INFO:teuthology.orchestra.run.smithi093.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.5 MB/s | 85 kB 00:00 2021-11-20T08:50:03.100 INFO:teuthology.orchestra.run.smithi093.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 803 kB/s | 23 kB 00:00 2021-11-20T08:50:03.102 INFO:teuthology.orchestra.run.smithi093.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2021-11-20T08:50:03.105 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:03.106 INFO:teuthology.orchestra.run.smithi093.stdout:Total 41 MB/s | 31 MB 00:00 2021-11-20T08:50:03.119 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:50:03.157 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:50:03.157 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:50:03.374 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:50:03.377 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:50:03.583 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:51 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:04.520 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:50:04.746 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T08:50:04.853 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:04.856 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:04.856 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:50:04.856 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:04.856 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:50:04.857 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T08:50:04.857 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:50:04.857 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T08:50:04.857 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T08:50:04.857 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T08:50:04.858 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T08:50:04.858 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T08:50:04.858 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T08:50:04.858 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T08:50:04.859 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T08:50:04.859 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T08:50:04.859 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T08:50:04.859 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T08:50:04.859 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T08:50:04.860 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T08:50:04.860 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T08:50:04.860 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T08:50:04.860 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T08:50:04.861 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T08:50:04.861 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T08:50:04.861 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T08:50:04.861 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T08:50:04.861 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T08:50:04.862 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T08:50:04.862 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T08:50:04.862 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T08:50:04.862 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T08:50:04.863 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T08:50:04.863 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T08:50:04.863 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T08:50:04.863 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T08:50:04.863 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T08:50:04.864 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T08:50:04.864 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:04.864 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:50:04.864 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:04.865 INFO:teuthology.orchestra.run.smithi102.stdout:Install 32 Packages 2021-11-20T08:50:04.865 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:04.866 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 31 M 2021-11-20T08:50:04.866 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 111 M 2021-11-20T08:50:04.866 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:50:04.881 INFO:teuthology.orchestra.run.smithi102.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 321 kB/s | 6.3 kB 00:00 2021-11-20T08:50:04.924 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T08:50:04.952 INFO:teuthology.orchestra.run.smithi102.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 27 MB/s | 2.4 MB 00:00 2021-11-20T08:50:04.961 INFO:teuthology.orchestra.run.smithi102.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2021-11-20T08:50:04.973 INFO:teuthology.orchestra.run.smithi102.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 16 MB/s | 220 kB 00:00 2021-11-20T08:50:05.011 INFO:teuthology.orchestra.run.smithi102.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 4.1 MB/s | 157 kB 00:00 2021-11-20T08:50:05.013 INFO:teuthology.orchestra.run.smithi102.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 25 MB/s | 39 kB 00:00 2021-11-20T08:50:05.015 INFO:teuthology.orchestra.run.smithi102.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 45 MB/s | 103 kB 00:00 2021-11-20T08:50:05.018 INFO:teuthology.orchestra.run.smithi102.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 28 MB/s | 54 kB 00:00 2021-11-20T08:50:05.024 INFO:teuthology.orchestra.run.smithi102.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 72 MB/s | 457 kB 00:00 2021-11-20T08:50:05.029 INFO:teuthology.orchestra.run.smithi102.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 56 MB/s | 237 kB 00:00 2021-11-20T08:50:05.038 INFO:teuthology.orchestra.run.smithi102.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 55 MB/s | 559 kB 00:00 2021-11-20T08:50:05.041 INFO:teuthology.orchestra.run.smithi102.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 51 MB/s | 109 kB 00:00 2021-11-20T08:50:05.097 INFO:teuthology.orchestra.run.smithi102.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2021-11-20T08:50:05.157 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T08:50:05.213 INFO:teuthology.orchestra.run.smithi102.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 255 kB/s | 44 kB 00:00 2021-11-20T08:50:05.371 INFO:teuthology.orchestra.run.smithi102.stdout:(15/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 43 MB/s | 18 MB 00:00 2021-11-20T08:50:05.372 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T08:50:05.374 INFO:teuthology.orchestra.run.smithi102.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 673 kB/s | 185 kB 00:00 2021-11-20T08:50:05.374 INFO:teuthology.orchestra.run.smithi102.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2021-11-20T08:50:05.402 INFO:teuthology.orchestra.run.smithi102.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 625 kB/s | 18 kB 00:00 2021-11-20T08:50:05.402 INFO:teuthology.orchestra.run.smithi102.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 379 kB/s | 11 kB 00:00 2021-11-20T08:50:05.431 INFO:teuthology.orchestra.run.smithi102.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T08:50:05.432 INFO:teuthology.orchestra.run.smithi102.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 1.6 MB/s | 49 kB 00:00 2021-11-20T08:50:05.461 INFO:teuthology.orchestra.run.smithi102.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 535 kB/s | 16 kB 00:00 2021-11-20T08:50:05.489 INFO:teuthology.orchestra.run.smithi102.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.8 MB/s | 283 kB 00:00 2021-11-20T08:50:05.490 INFO:teuthology.orchestra.run.smithi102.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 657 kB/s | 19 kB 00:00 2021-11-20T08:50:05.504 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T08:50:05.519 INFO:teuthology.orchestra.run.smithi102.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 860 kB/s | 24 kB 00:00 2021-11-20T08:50:05.519 INFO:teuthology.orchestra.run.smithi102.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-20T08:50:05.545 INFO:teuthology.orchestra.run.smithi102.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 977 kB/s | 27 kB 00:00 2021-11-20T08:50:05.576 INFO:teuthology.orchestra.run.smithi102.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 8.0 MB/s | 251 kB 00:00 2021-11-20T08:50:05.577 INFO:teuthology.orchestra.run.smithi102.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2021-11-20T08:50:05.605 INFO:teuthology.orchestra.run.smithi102.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2021-11-20T08:50:05.607 INFO:teuthology.orchestra.run.smithi102.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 800 kB/s | 23 kB 00:00 2021-11-20T08:50:05.620 INFO:teuthology.orchestra.run.smithi102.stdout:(32/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2021-11-20T08:50:05.624 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:05.624 INFO:teuthology.orchestra.run.smithi102.stdout:Total 40 MB/s | 31 MB 00:00 2021-11-20T08:50:05.637 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:50:05.659 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T08:50:05.675 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:50:05.675 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:50:05.868 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T08:50:05.892 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:50:05.895 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:50:06.047 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T08:50:06.275 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T08:50:06.487 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T08:50:06.742 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T08:50:06.940 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T08:50:06.995 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:50:07.226 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T08:50:07.373 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T08:50:07.450 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T08:50:07.603 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T08:50:07.666 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T08:50:07.832 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T08:50:07.848 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T08:50:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T08:50:08.067 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T08:50:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T08:50:08.300 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T08:50:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T08:50:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T08:50:08.612 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T08:50:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T08:50:08.813 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T08:50:08.960 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T08:50:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T08:50:09.179 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T08:50:09.200 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T08:50:09.407 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T08:50:09.456 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T08:50:09.695 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T08:50:09.777 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T08:50:09.905 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T08:50:10.020 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T08:50:10.216 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T08:50:10.271 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T08:50:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T08:50:10.585 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T08:50:10.668 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T08:50:10.907 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T08:50:10.981 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T08:50:11.111 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T08:50:11.199 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T08:50:11.352 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T08:50:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T08:50:11.766 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T08:50:11.987 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T08:50:12.232 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T08:50:12.595 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T08:50:12.634 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T08:50:12.836 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T08:50:12.919 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T08:50:13.247 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T08:50:13.267 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T08:50:13.425 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T08:50:13.511 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T08:50:13.733 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T08:50:13.948 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T08:50:13.965 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T08:50:14.089 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T08:50:14.487 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T08:50:14.488 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T08:50:14.488 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T08:50:14.488 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T08:50:14.488 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T08:50:14.489 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T08:50:14.490 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T08:50:14.490 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T08:50:14.491 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T08:50:14.491 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T08:50:14.491 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T08:50:14.492 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T08:50:14.492 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T08:50:14.492 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T08:50:14.493 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T08:50:14.493 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T08:50:14.494 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T08:50:14.494 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T08:50:14.494 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T08:50:14.495 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T08:50:14.495 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T08:50:14.496 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T08:50:14.496 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T08:50:14.496 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T08:50:14.497 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T08:50:14.497 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T08:50:14.498 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T08:50:14.498 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T08:50:14.499 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T08:50:14.500 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T08:50:14.500 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T08:50:14.501 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T08:50:14.947 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T08:50:15.167 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T08:50:15.314 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T08:50:15.314 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:15.315 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:15.315 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:15.315 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:15.315 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:15.316 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T08:50:15.316 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:15.316 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:15.316 INFO:teuthology.orchestra.run.smithi093.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T08:50:15.316 INFO:teuthology.orchestra.run.smithi093.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T08:50:15.317 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T08:50:15.317 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T08:50:15.317 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T08:50:15.317 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T08:50:15.318 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T08:50:15.318 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T08:50:15.318 INFO:teuthology.orchestra.run.smithi093.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T08:50:15.318 INFO:teuthology.orchestra.run.smithi093.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T08:50:15.319 INFO:teuthology.orchestra.run.smithi093.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T08:50:15.319 INFO:teuthology.orchestra.run.smithi093.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T08:50:15.319 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T08:50:15.319 INFO:teuthology.orchestra.run.smithi093.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T08:50:15.320 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T08:50:15.320 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T08:50:15.320 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T08:50:15.320 INFO:teuthology.orchestra.run.smithi093.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T08:50:15.321 INFO:teuthology.orchestra.run.smithi093.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T08:50:15.321 INFO:teuthology.orchestra.run.smithi093.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T08:50:15.321 INFO:teuthology.orchestra.run.smithi093.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T08:50:15.321 INFO:teuthology.orchestra.run.smithi093.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T08:50:15.321 INFO:teuthology.orchestra.run.smithi093.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T08:50:15.322 INFO:teuthology.orchestra.run.smithi093.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T08:50:15.322 INFO:teuthology.orchestra.run.smithi093.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T08:50:15.322 INFO:teuthology.orchestra.run.smithi093.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T08:50:15.322 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:15.569 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-base 2021-11-20T08:50:15.599 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T08:50:15.825 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T08:50:16.047 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T08:50:16.109 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:04 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:16.233 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T08:50:16.249 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T08:50:16.381 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T08:50:16.755 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T08:50:16.755 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T08:50:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T08:50:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T08:50:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T08:50:16.756 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T08:50:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T08:50:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T08:50:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T08:50:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T08:50:16.757 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T08:50:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T08:50:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T08:50:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T08:50:16.758 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T08:50:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T08:50:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T08:50:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T08:50:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T08:50:16.759 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T08:50:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T08:50:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T08:50:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T08:50:16.760 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T08:50:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T08:50:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T08:50:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T08:50:16.761 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T08:50:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T08:50:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T08:50:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T08:50:16.762 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T08:50:17.110 INFO:teuthology.orchestra.run.smithi093.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:50:17.399 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:17.399 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:50:17.400 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:17.468 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T08:50:17.468 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:17.469 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:50:17.469 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:17.470 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:17.470 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:17.470 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T08:50:17.470 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:17.471 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:17.471 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T08:50:17.471 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T08:50:17.471 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T08:50:17.471 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T08:50:17.472 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T08:50:17.472 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T08:50:17.472 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T08:50:17.473 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T08:50:17.473 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T08:50:17.473 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T08:50:17.473 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T08:50:17.474 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T08:50:17.474 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T08:50:17.474 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T08:50:17.474 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T08:50:17.474 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T08:50:17.475 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T08:50:17.476 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T08:50:17.476 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T08:50:17.477 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T08:50:17.477 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T08:50:17.477 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T08:50:17.478 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T08:50:17.478 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T08:50:17.479 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T08:50:17.479 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T08:50:17.479 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:17.480 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:17.485 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install cephadm 2021-11-20T08:50:17.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2021-11-20T08:50:18.023 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:06 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:18.264 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:06 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:19.242 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:50:19.288 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:19.289 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:19.290 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:50:19.290 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:19.290 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:50:19.290 INFO:teuthology.orchestra.run.smithi093.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T08:50:19.291 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:19.291 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:50:19.291 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:19.291 INFO:teuthology.orchestra.run.smithi093.stdout:Install 1 Package 2021-11-20T08:50:19.292 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:19.292 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 71 k 2021-11-20T08:50:19.292 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 290 k 2021-11-20T08:50:19.292 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:50:19.302 INFO:teuthology.orchestra.run.smithi093.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.9 MB/s | 71 kB 00:00 2021-11-20T08:50:19.303 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:19.303 INFO:teuthology.orchestra.run.smithi093.stdout:Total 5.3 MB/s | 71 kB 00:00 2021-11-20T08:50:19.304 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:50:19.311 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:50:19.311 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:50:19.312 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:50:19.316 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:50:19.359 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:50:19.529 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:19.531 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:50:19.531 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:19.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2021-11-20T08:50:20.120 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:08 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:21.379 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:21.380 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:21.380 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:50:21.381 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:21.381 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:50:21.382 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T08:50:21.382 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:21.382 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:50:21.383 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:21.383 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-11-20T08:50:21.383 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:21.384 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 71 k 2021-11-20T08:50:21.384 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 290 k 2021-11-20T08:50:21.384 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:50:21.399 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.8 MB/s | 71 kB 00:00 2021-11-20T08:50:21.400 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:21.401 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.2 MB/s | 71 kB 00:00 2021-11-20T08:50:21.401 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:50:21.402 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:50:21.403 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:50:21.404 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:50:21.406 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:50:21.451 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:50:22.163 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:22.464 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:22.660 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:23.008 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:23.008 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:23.009 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:23.009 INFO:teuthology.orchestra.run.smithi093.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T08:50:23.009 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:23.010 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:23.181 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T08:50:23.732 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:11 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:24.122 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:24.371 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:24.566 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:24.888 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T08:50:24.888 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:24.889 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:50:24.889 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T08:50:24.889 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:24.889 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:25.046 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:25.047 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:25.047 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repo Size 2021-11-20T08:50:25.047 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:25.047 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:50:25.048 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T08:50:25.048 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:25.048 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:50:25.048 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:25.049 INFO:teuthology.orchestra.run.smithi093.stdout:Install 1 Package 2021-11-20T08:50:25.049 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:25.049 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 163 k 2021-11-20T08:50:25.049 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 587 k 2021-11-20T08:50:25.050 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:50:25.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T08:50:25.060 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 12 MB/s | 163 kB 00:00 2021-11-20T08:50:25.061 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:25.061 INFO:teuthology.orchestra.run.smithi093.stdout:Total 11 MB/s | 163 kB 00:00 2021-11-20T08:50:25.062 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:50:25.074 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:50:25.075 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:50:25.122 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:50:25.125 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:50:25.249 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:50:25.438 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:50:25.607 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:13 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:25.820 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:50:26.189 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:50:26.189 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:26.190 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:26.190 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:26.190 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:26.191 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:26.370 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-mgr 2021-11-20T08:50:26.912 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:26.914 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:26.914 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2021-11-20T08:50:26.914 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:26.914 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:50:26.915 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T08:50:26.915 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:26.915 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:50:26.915 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:26.916 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-11-20T08:50:26.916 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:26.916 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 163 k 2021-11-20T08:50:26.916 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 587 k 2021-11-20T08:50:26.917 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:50:26.923 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:14 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:26.927 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 12 MB/s | 163 kB 00:00 2021-11-20T08:50:26.928 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:26.929 INFO:teuthology.orchestra.run.smithi102.stdout:Total 11 MB/s | 163 kB 00:00 2021-11-20T08:50:26.929 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:50:26.941 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:50:26.941 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:50:26.988 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:50:26.992 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:50:27.101 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:50:27.306 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:50:27.690 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:50:28.005 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:50:28.005 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:28.006 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:50:28.006 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T08:50:28.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:28.007 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:28.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2021-11-20T08:50:28.761 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:16 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:28.893 INFO:teuthology.orchestra.run.smithi093.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:50:29.179 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:29.180 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:50:29.181 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:29.267 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T08:50:29.801 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:17 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:29.825 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:50:30.112 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:30.114 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:50:30.114 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:30.183 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T08:50:30.721 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:18 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:31.136 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:31.140 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:31.141 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:50:31.141 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:31.142 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:50:31.143 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T08:50:31.143 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:50:31.144 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T08:50:31.145 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T08:50:31.145 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T08:50:31.145 INFO:teuthology.orchestra.run.smithi093.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T08:50:31.146 INFO:teuthology.orchestra.run.smithi093.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T08:50:31.147 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:31.148 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:50:31.148 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:31.150 INFO:teuthology.orchestra.run.smithi093.stdout:Install 6 Packages 2021-11-20T08:50:31.150 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:31.151 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 3.0 M 2021-11-20T08:50:31.151 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 15 M 2021-11-20T08:50:31.151 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:50:31.161 INFO:teuthology.orchestra.run.smithi093.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.1 MB/s | 24 kB 00:00 2021-11-20T08:50:31.167 INFO:teuthology.orchestra.run.smithi093.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 324 kB/s | 8.9 kB 00:00 2021-11-20T08:50:31.194 INFO:teuthology.orchestra.run.smithi093.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 49 MB/s | 2.7 MB 00:00 2021-11-20T08:50:31.197 INFO:teuthology.orchestra.run.smithi093.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2021-11-20T08:50:31.251 INFO:teuthology.orchestra.run.smithi093.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 403 kB/s | 34 kB 00:00 2021-11-20T08:50:31.337 INFO:teuthology.orchestra.run.smithi093.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2021-11-20T08:50:31.340 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:31.341 INFO:teuthology.orchestra.run.smithi093.stdout:Total 15 MB/s | 3.0 MB 00:00 2021-11-20T08:50:31.343 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:50:31.351 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:50:31.352 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:50:31.422 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:50:31.426 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:50:31.683 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:50:31.938 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T08:50:32.007 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:32.011 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:32.011 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:50:32.012 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:32.012 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:50:32.013 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T08:50:32.013 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:50:32.014 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T08:50:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T08:50:32.015 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T08:50:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T08:50:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T08:50:32.016 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:32.017 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:50:32.017 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:32.017 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2021-11-20T08:50:32.018 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:32.018 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.0 M 2021-11-20T08:50:32.019 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 15 M 2021-11-20T08:50:32.019 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:50:32.032 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.2 MB/s | 24 kB 00:00 2021-11-20T08:50:32.033 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 408 kB/s | 8.9 kB 00:00 2021-11-20T08:50:32.066 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 49 MB/s | 2.7 MB 00:00 2021-11-20T08:50:32.068 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2021-11-20T08:50:32.122 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 400 kB/s | 34 kB 00:00 2021-11-20T08:50:32.148 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T08:50:32.206 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-11-20T08:50:32.209 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:32.209 INFO:teuthology.orchestra.run.smithi102.stdout:Total 15 MB/s | 3.0 MB 00:00 2021-11-20T08:50:32.211 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:50:32.220 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:50:32.220 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:50:32.291 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:50:32.294 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:50:32.356 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T08:50:32.476 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T08:50:32.542 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:50:32.804 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T08:50:33.058 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T08:50:33.249 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T08:50:33.361 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T08:50:33.539 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T08:50:34.049 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:50:34.327 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:50:34.328 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T08:50:34.330 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T08:50:34.330 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T08:50:34.330 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T08:50:34.331 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T08:50:34.387 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T08:50:34.669 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T08:50:34.669 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:34.669 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:34.670 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T08:50:34.670 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T08:50:34.670 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T08:50:34.670 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T08:50:34.671 INFO:teuthology.orchestra.run.smithi093.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T08:50:34.671 INFO:teuthology.orchestra.run.smithi093.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T08:50:34.671 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:34.671 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:34.767 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T08:50:35.305 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:23 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:35.697 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:50:35.966 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:50:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T08:50:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T08:50:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T08:50:35.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T08:50:35.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T08:50:36.285 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T08:50:36.285 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:36.286 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:50:36.286 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T08:50:36.286 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T08:50:36.286 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T08:50:36.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T08:50:36.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T08:50:36.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T08:50:36.287 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:36.288 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:36.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T08:50:36.581 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:36.583 INFO:teuthology.orchestra.run.smithi093.stdout:===================================================================================================== 2021-11-20T08:50:36.584 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:50:36.584 INFO:teuthology.orchestra.run.smithi093.stdout:===================================================================================================== 2021-11-20T08:50:36.584 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:50:36.584 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T08:50:36.585 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:50:36.585 INFO:teuthology.orchestra.run.smithi093.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T08:50:36.585 INFO:teuthology.orchestra.run.smithi093.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T08:50:36.585 INFO:teuthology.orchestra.run.smithi093.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T08:50:36.586 INFO:teuthology.orchestra.run.smithi093.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T08:50:36.586 INFO:teuthology.orchestra.run.smithi093.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T08:50:36.586 INFO:teuthology.orchestra.run.smithi093.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T08:50:36.586 INFO:teuthology.orchestra.run.smithi093.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T08:50:36.586 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:36.587 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:50:36.587 INFO:teuthology.orchestra.run.smithi093.stdout:===================================================================================================== 2021-11-20T08:50:36.587 INFO:teuthology.orchestra.run.smithi093.stdout:Install 8 Packages 2021-11-20T08:50:36.587 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:36.588 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 37 M 2021-11-20T08:50:36.588 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 211 M 2021-11-20T08:50:36.588 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:50:36.703 INFO:teuthology.orchestra.run.smithi093.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 73 MB/s | 8.6 MB 00:00 2021-11-20T08:50:36.734 INFO:teuthology.orchestra.run.smithi093.stdout:(2/8): openblas-0.3.15-3.el8.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2021-11-20T08:50:36.739 INFO:teuthology.orchestra.run.smithi093.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 35 MB/s | 225 kB 00:00 2021-11-20T08:50:36.772 INFO:teuthology.orchestra.run.smithi093.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 26 MB/s | 4.9 MB 00:00 2021-11-20T08:50:36.790 INFO:teuthology.orchestra.run.smithi093.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 36 MB/s | 643 kB 00:00 2021-11-20T08:50:36.799 INFO:teuthology.orchestra.run.smithi093.stdout:(6/8): libquadmath-8.5.0-3.el8.x86_64.rpm 20 MB/s | 170 kB 00:00 2021-11-20T08:50:36.902 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:24 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:37.012 INFO:teuthology.orchestra.run.smithi093.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 50 MB/s | 14 MB 00:00 2021-11-20T08:50:37.032 INFO:teuthology.orchestra.run.smithi093.stdout:(8/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-11-20T08:50:37.037 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:37.037 INFO:teuthology.orchestra.run.smithi093.stdout:Total 81 MB/s | 37 MB 00:00 2021-11-20T08:50:37.051 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:50:37.070 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:50:37.070 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:50:37.374 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:50:37.377 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:50:37.947 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:50:38.111 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T08:50:38.158 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:38.159 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-11-20T08:50:38.159 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:50:38.159 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-11-20T08:50:38.160 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:50:38.160 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T08:50:38.160 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:50:38.160 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T08:50:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T08:50:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T08:50:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T08:50:38.161 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T08:50:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T08:50:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T08:50:38.162 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:38.162 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:50:38.163 INFO:teuthology.orchestra.run.smithi102.stdout:===================================================================================================== 2021-11-20T08:50:38.163 INFO:teuthology.orchestra.run.smithi102.stdout:Install 8 Packages 2021-11-20T08:50:38.163 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:38.164 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 37 M 2021-11-20T08:50:38.164 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 211 M 2021-11-20T08:50:38.164 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:50:38.236 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T08:50:38.280 INFO:teuthology.orchestra.run.smithi102.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 73 MB/s | 8.6 MB 00:00 2021-11-20T08:50:38.293 INFO:teuthology.orchestra.run.smithi102.stdout:(2/8): openblas-0.3.15-3.el8.x86_64.rpm 36 MB/s | 4.7 MB 00:00 2021-11-20T08:50:38.304 INFO:teuthology.orchestra.run.smithi102.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 39 MB/s | 225 kB 00:00 2021-11-20T08:50:38.355 INFO:teuthology.orchestra.run.smithi102.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 25 MB/s | 4.9 MB 00:00 2021-11-20T08:50:38.370 INFO:teuthology.orchestra.run.smithi102.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 45 MB/s | 643 kB 00:00 2021-11-20T08:50:38.375 INFO:teuthology.orchestra.run.smithi102.stdout:(6/8): libquadmath-8.5.0-3.el8.x86_64.rpm 31 MB/s | 170 kB 00:00 2021-11-20T08:50:38.386 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T08:50:38.469 INFO:teuthology.orchestra.run.smithi102.stdout:(7/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2021-11-20T08:50:38.610 INFO:teuthology.orchestra.run.smithi102.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 45 MB/s | 14 MB 00:00 2021-11-20T08:50:38.616 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:38.616 INFO:teuthology.orchestra.run.smithi102.stdout:Total 81 MB/s | 37 MB 00:00 2021-11-20T08:50:38.629 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:50:38.648 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:50:38.649 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:50:38.949 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:50:38.953 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:50:38.996 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T08:50:39.153 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T08:50:39.488 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:50:39.637 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T08:50:39.766 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T08:50:39.767 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T08:50:39.904 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T08:50:39.927 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T08:50:40.500 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T08:50:40.654 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T08:50:40.655 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T08:50:41.011 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T08:50:41.285 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T08:50:41.297 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T08:50:42.131 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T08:50:42.847 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T08:50:43.089 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T08:50:43.171 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T08:50:43.429 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T08:50:44.408 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T08:50:44.677 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T08:50:45.227 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T08:50:45.263 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T08:50:45.263 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T08:50:45.264 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T08:50:45.264 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T08:50:45.266 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T08:50:45.267 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T08:50:45.268 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T08:50:45.268 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T08:50:46.195 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T08:50:46.195 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:46.195 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:46.195 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T08:50:46.196 INFO:teuthology.orchestra.run.smithi093.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T08:50:46.196 INFO:teuthology.orchestra.run.smithi093.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T08:50:46.196 INFO:teuthology.orchestra.run.smithi093.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T08:50:46.196 INFO:teuthology.orchestra.run.smithi093.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T08:50:46.197 INFO:teuthology.orchestra.run.smithi093.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T08:50:46.197 INFO:teuthology.orchestra.run.smithi093.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T08:50:46.197 INFO:teuthology.orchestra.run.smithi093.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T08:50:46.197 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:46.197 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:46.299 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-mgr-rook 2021-11-20T08:50:46.505 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T08:50:46.671 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T08:50:46.838 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:34 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T08:50:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T08:50:46.936 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T08:50:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T08:50:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T08:50:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T08:50:46.937 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T08:50:46.938 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T08:50:47.804 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T08:50:47.805 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:47.805 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:50:47.805 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T08:50:47.806 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T08:50:47.806 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T08:50:47.806 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T08:50:47.806 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T08:50:47.807 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T08:50:47.807 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T08:50:47.807 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T08:50:47.807 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:47.808 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:48.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2021-11-20T08:50:48.135 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:48.137 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:48.137 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:50:48.137 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:48.138 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:50:48.138 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T08:50:48.139 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:50:48.139 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T08:50:48.139 INFO:teuthology.orchestra.run.smithi093.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T08:50:48.139 INFO:teuthology.orchestra.run.smithi093.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T08:50:48.140 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T08:50:48.140 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T08:50:48.140 INFO:teuthology.orchestra.run.smithi093.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T08:50:48.141 INFO:teuthology.orchestra.run.smithi093.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T08:50:48.142 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T08:50:48.142 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T08:50:48.142 INFO:teuthology.orchestra.run.smithi093.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T08:50:48.143 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T08:50:48.143 INFO:teuthology.orchestra.run.smithi093.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T08:50:48.143 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:48.143 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:50:48.143 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:48.144 INFO:teuthology.orchestra.run.smithi093.stdout:Install 13 Packages 2021-11-20T08:50:48.144 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:48.146 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 1.9 M 2021-11-20T08:50:48.146 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 24 M 2021-11-20T08:50:48.147 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:50:48.159 INFO:teuthology.orchestra.run.smithi093.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-11-20T08:50:48.176 INFO:teuthology.orchestra.run.smithi093.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 736 kB/s | 27 kB 00:00 2021-11-20T08:50:48.179 INFO:teuthology.orchestra.run.smithi093.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 41 MB/s | 110 kB 00:00 2021-11-20T08:50:48.181 INFO:teuthology.orchestra.run.smithi093.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 489 kB/s | 20 kB 00:00 2021-11-20T08:50:48.184 INFO:teuthology.orchestra.run.smithi093.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 55 MB/s | 155 kB 00:00 2021-11-20T08:50:48.184 INFO:teuthology.orchestra.run.smithi093.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 30 MB/s | 43 kB 00:00 2021-11-20T08:50:48.189 INFO:teuthology.orchestra.run.smithi093.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.2 MB/s | 126 kB 00:00 2021-11-20T08:50:48.271 INFO:teuthology.orchestra.run.smithi093.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 191 kB/s | 16 kB 00:00 2021-11-20T08:50:48.272 INFO:teuthology.orchestra.run.smithi093.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 381 kB/s | 33 kB 00:00 2021-11-20T08:50:48.300 INFO:teuthology.orchestra.run.smithi093.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 732 kB/s | 81 kB 00:00 2021-11-20T08:50:48.326 INFO:teuthology.orchestra.run.smithi093.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2021-11-20T08:50:48.328 INFO:teuthology.orchestra.run.smithi093.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-11-20T08:50:48.470 INFO:teuthology.orchestra.run.smithi093.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.8 MB/s | 1.2 MB 00:00 2021-11-20T08:50:48.473 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:48.473 INFO:teuthology.orchestra.run.smithi093.stdout:Total 5.7 MB/s | 1.9 MB 00:00 2021-11-20T08:50:48.484 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:50:48.497 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:50:48.497 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:50:48.574 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:50:48.577 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:50:48.632 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:36 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:49.093 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:50:49.323 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T08:50:49.512 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T08:50:49.755 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T08:50:49.912 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:49.915 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:49.915 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:50:49.915 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:49.916 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:50:49.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T08:50:49.917 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:50:49.917 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T08:50:49.917 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T08:50:49.918 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T08:50:49.918 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T08:50:49.918 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T08:50:49.918 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T08:50:49.919 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T08:50:49.919 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T08:50:49.919 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T08:50:49.919 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T08:50:49.919 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T08:50:49.920 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T08:50:49.920 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:49.921 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:50:49.921 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:49.921 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2021-11-20T08:50:49.921 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:49.922 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 1.9 M 2021-11-20T08:50:49.922 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 24 M 2021-11-20T08:50:49.922 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:50:49.936 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-11-20T08:50:49.950 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T08:50:49.953 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 746 kB/s | 27 kB 00:00 2021-11-20T08:50:49.956 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 35 MB/s | 110 kB 00:00 2021-11-20T08:50:49.957 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 506 kB/s | 20 kB 00:00 2021-11-20T08:50:49.958 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 29 MB/s | 43 kB 00:00 2021-11-20T08:50:49.959 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 48 MB/s | 155 kB 00:00 2021-11-20T08:50:49.968 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.9 MB/s | 126 kB 00:00 2021-11-20T08:50:50.043 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 393 kB/s | 33 kB 00:00 2021-11-20T08:50:50.045 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 192 kB/s | 16 kB 00:00 2021-11-20T08:50:50.079 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 731 kB/s | 81 kB 00:00 2021-11-20T08:50:50.102 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-20T08:50:50.108 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-11-20T08:50:50.141 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T08:50:50.240 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2021-11-20T08:50:50.243 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:50.243 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.8 MB/s | 1.9 MB 00:00 2021-11-20T08:50:50.253 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:50:50.267 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:50:50.267 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:50:50.343 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:50:50.346 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:50:50.400 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T08:50:50.660 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T08:50:50.828 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:50:50.898 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T08:50:51.016 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T08:50:51.223 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T08:50:51.449 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T08:50:51.643 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T08:50:51.844 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T08:50:51.977 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T08:50:52.070 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T08:50:52.242 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T08:50:52.329 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T08:50:52.391 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T08:50:52.543 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T08:50:52.550 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T08:50:52.743 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T08:50:53.002 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T08:50:53.003 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T08:50:53.003 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T08:50:53.004 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T08:50:53.005 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T08:50:53.005 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T08:50:53.006 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T08:50:53.006 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T08:50:53.009 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T08:50:53.009 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T08:50:53.009 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T08:50:53.009 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T08:50:53.010 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T08:50:53.427 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T08:50:53.427 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:53.427 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:53.428 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T08:50:53.428 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T08:50:53.429 INFO:teuthology.orchestra.run.smithi093.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T08:50:53.429 INFO:teuthology.orchestra.run.smithi093.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T08:50:53.429 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T08:50:53.429 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T08:50:53.430 INFO:teuthology.orchestra.run.smithi093.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T08:50:53.430 INFO:teuthology.orchestra.run.smithi093.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T08:50:53.430 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T08:50:53.430 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T08:50:53.431 INFO:teuthology.orchestra.run.smithi093.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T08:50:53.431 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T08:50:53.431 INFO:teuthology.orchestra.run.smithi093.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T08:50:53.431 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:53.431 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:53.545 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T08:50:53.621 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T08:50:53.893 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T08:50:54.076 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T08:50:54.082 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:42 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:54.243 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T08:50:54.394 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T08:50:54.650 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T08:50:54.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T08:50:54.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T08:50:54.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T08:50:54.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T08:50:54.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T08:50:54.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T08:50:54.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T08:50:54.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T08:50:54.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T08:50:54.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T08:50:54.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T08:50:54.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T08:50:55.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T08:50:55.035 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:55.035 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:50:55.035 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T08:50:55.036 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T08:50:55.036 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T08:50:55.036 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T08:50:55.037 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T08:50:55.037 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T08:50:55.037 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T08:50:55.037 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T08:50:55.038 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T08:50:55.038 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T08:50:55.038 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T08:50:55.038 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T08:50:55.038 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T08:50:55.039 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:55.039 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:50:55.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T08:50:55.379 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:50:55.380 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:55.380 INFO:teuthology.orchestra.run.smithi093.stdout: Package Arch Version Repository Size 2021-11-20T08:50:55.380 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:55.380 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:50:55.381 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T08:50:55.381 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:50:55.381 INFO:teuthology.orchestra.run.smithi093.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T08:50:55.382 INFO:teuthology.orchestra.run.smithi093.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T08:50:55.382 INFO:teuthology.orchestra.run.smithi093.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T08:50:55.382 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T08:50:55.382 INFO:teuthology.orchestra.run.smithi093.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T08:50:55.383 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:55.383 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:50:55.383 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:50:55.383 INFO:teuthology.orchestra.run.smithi093.stdout:Install 6 Packages 2021-11-20T08:50:55.384 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:55.385 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 5.8 M 2021-11-20T08:50:55.385 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 24 M 2021-11-20T08:50:55.385 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:50:55.402 INFO:teuthology.orchestra.run.smithi093.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.4 MB/s | 89 kB 00:00 2021-11-20T08:50:55.449 INFO:teuthology.orchestra.run.smithi093.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 7.9 MB/s | 538 kB 00:00 2021-11-20T08:50:55.487 INFO:teuthology.orchestra.run.smithi093.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 250 kB/s | 21 kB 00:00 2021-11-20T08:50:55.542 INFO:teuthology.orchestra.run.smithi093.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 887 kB/s | 49 kB 00:00 2021-11-20T08:50:55.582 INFO:teuthology.orchestra.run.smithi093.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2021-11-20T08:50:55.612 INFO:teuthology.orchestra.run.smithi093.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2021-11-20T08:50:55.615 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:55.615 INFO:teuthology.orchestra.run.smithi093.stdout:Total 25 MB/s | 5.8 MB 00:00 2021-11-20T08:50:55.620 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:50:55.629 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:50:55.629 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:50:55.670 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:43 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:50:55.712 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:50:55.716 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:50:55.977 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:50:56.263 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T08:50:56.779 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T08:50:56.936 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:50:56.937 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:56.937 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2021-11-20T08:50:56.937 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:56.938 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:50:56.938 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T08:50:56.939 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:50:56.939 INFO:teuthology.orchestra.run.smithi102.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T08:50:56.939 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T08:50:56.941 INFO:teuthology.orchestra.run.smithi102.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T08:50:56.941 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T08:50:56.941 INFO:teuthology.orchestra.run.smithi102.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T08:50:56.942 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:56.942 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:50:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:50:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2021-11-20T08:50:56.944 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:50:56.946 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 5.8 M 2021-11-20T08:50:56.947 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 24 M 2021-11-20T08:50:56.947 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:50:56.962 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.0 MB/s | 89 kB 00:00 2021-11-20T08:50:57.005 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 8.2 MB/s | 538 kB 00:00 2021-11-20T08:50:57.037 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2021-11-20T08:50:57.043 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 259 kB/s | 21 kB 00:00 2021-11-20T08:50:57.148 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 442 kB/s | 49 kB 00:00 2021-11-20T08:50:57.178 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.0 MB/s | 355 kB 00:00 2021-11-20T08:50:57.180 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:50:57.181 INFO:teuthology.orchestra.run.smithi102.stdout:Total 24 MB/s | 5.8 MB 00:00 2021-11-20T08:50:57.185 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:50:57.194 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:50:57.194 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:50:57.278 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:50:57.281 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:50:57.570 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:50:57.702 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T08:50:57.816 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T08:50:58.049 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T08:50:58.069 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T08:50:58.307 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T08:50:58.541 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:50:58.792 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:50:58.792 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T08:50:58.793 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T08:50:58.793 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T08:50:58.793 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T08:50:58.793 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T08:50:58.989 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T08:50:59.124 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T08:50:59.124 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:59.125 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:50:59.125 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T08:50:59.125 INFO:teuthology.orchestra.run.smithi093.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T08:50:59.125 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T08:50:59.126 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:50:59.126 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:50:59.235 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install ceph-fuse 2021-11-20T08:50:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T08:50:59.591 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T08:50:59.760 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:50:59.774 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:47 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:00.015 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T08:51:00.015 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T08:51:00.015 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T08:51:00.016 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T08:51:00.016 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T08:51:00.016 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T08:51:00.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T08:51:00.349 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:00.349 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:51:00.349 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T08:51:00.350 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T08:51:00.350 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T08:51:00.350 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:00.350 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:00.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2021-11-20T08:51:00.984 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:48 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:01.051 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:01.052 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:01.053 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:01.053 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:01.053 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:51:01.053 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T08:51:01.054 INFO:teuthology.orchestra.run.smithi093.stdout:Installing dependencies: 2021-11-20T08:51:01.054 INFO:teuthology.orchestra.run.smithi093.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T08:51:01.054 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:01.054 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:51:01.055 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:01.055 INFO:teuthology.orchestra.run.smithi093.stdout:Install 2 Packages 2021-11-20T08:51:01.055 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:01.055 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 967 k 2021-11-20T08:51:01.055 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 3.2 M 2021-11-20T08:51:01.056 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:51:01.078 INFO:teuthology.orchestra.run.smithi093.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 35 MB/s | 885 kB 00:00 2021-11-20T08:51:01.089 INFO:teuthology.orchestra.run.smithi093.stdout:(2/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-11-20T08:51:01.091 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:01.092 INFO:teuthology.orchestra.run.smithi093.stdout:Total 24 MB/s | 967 kB 00:00 2021-11-20T08:51:01.093 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:51:01.115 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:51:01.116 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:51:01.196 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:51:01.199 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:51:01.361 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:51:01.667 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T08:51:01.857 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T08:51:02.227 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T08:51:02.227 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T08:51:02.319 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:02.320 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:02.320 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:02.321 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:02.321 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:51:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T08:51:02.321 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2021-11-20T08:51:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T08:51:02.322 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:02.322 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:51:02.323 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:02.323 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2021-11-20T08:51:02.323 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:02.323 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 967 k 2021-11-20T08:51:02.324 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 3.2 M 2021-11-20T08:51:02.324 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:51:02.346 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 35 MB/s | 885 kB 00:00 2021-11-20T08:51:02.358 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-11-20T08:51:02.359 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:02.360 INFO:teuthology.orchestra.run.smithi102.stdout:Total 24 MB/s | 967 kB 00:00 2021-11-20T08:51:02.362 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:51:02.384 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:51:02.385 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:51:02.466 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:51:02.469 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:51:02.551 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T08:51:02.551 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:02.552 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:51:02.552 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T08:51:02.553 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:02.554 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:02.647 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:51:02.726 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install librados-devel 2021-11-20T08:51:02.943 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T08:51:03.118 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T08:51:03.278 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:51 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:03.468 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T08:51:03.468 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T08:51:03.776 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T08:51:03.776 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:03.776 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:51:03.777 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T08:51:03.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:03.777 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:03.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2021-11-20T08:51:04.516 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:52 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:04.615 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:04.616 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:04.616 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:04.616 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:04.617 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:51:04.617 INFO:teuthology.orchestra.run.smithi093.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T08:51:04.617 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:04.617 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:51:04.618 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:04.618 INFO:teuthology.orchestra.run.smithi093.stdout:Install 1 Package 2021-11-20T08:51:04.618 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:04.619 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 131 k 2021-11-20T08:51:04.619 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 439 k 2021-11-20T08:51:04.619 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:51:04.629 INFO:teuthology.orchestra.run.smithi093.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 10 MB/s | 131 kB 00:00 2021-11-20T08:51:04.629 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:04.630 INFO:teuthology.orchestra.run.smithi093.stdout:Total 9.3 MB/s | 131 kB 00:00 2021-11-20T08:51:04.630 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:51:04.637 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:51:04.638 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:51:04.688 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:51:04.692 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:51:04.829 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:51:05.064 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:05.324 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:05.690 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:05.690 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:05.690 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:51:05.690 INFO:teuthology.orchestra.run.smithi093.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:05.691 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:05.691 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:05.793 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:05.794 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:05.795 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:05.795 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:05.795 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:51:05.795 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T08:51:05.796 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:05.796 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:51:05.796 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:05.796 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-11-20T08:51:05.797 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:05.797 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 131 k 2021-11-20T08:51:05.797 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 439 k 2021-11-20T08:51:05.797 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:51:05.807 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 11 MB/s | 131 kB 00:00 2021-11-20T08:51:05.807 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:05.808 INFO:teuthology.orchestra.run.smithi102.stdout:Total 9.5 MB/s | 131 kB 00:00 2021-11-20T08:51:05.808 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:51:05.815 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:51:05.815 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:51:05.866 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:51:05.869 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:51:05.876 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install libcephfs2 2021-11-20T08:51:05.980 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:51:06.200 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:06.420 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:54 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:06.456 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:06.798 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:06.799 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:06.799 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:51:06.800 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:06.801 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:06.801 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:06.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2021-11-20T08:51:07.503 INFO:teuthology.orchestra.run.smithi093.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:07.519 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:07.796 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:07.797 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:51:07.797 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:07.878 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install libcephfs-devel 2021-11-20T08:51:08.416 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:56 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:08.527 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:08.822 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:08.823 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:51:08.823 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:08.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2021-11-20T08:51:09.430 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:57 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:09.702 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:09.704 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:09.704 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:09.705 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:09.705 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:51:09.705 INFO:teuthology.orchestra.run.smithi093.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T08:51:09.706 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:09.706 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:51:09.707 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:09.707 INFO:teuthology.orchestra.run.smithi093.stdout:Install 1 Package 2021-11-20T08:51:09.708 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:09.708 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 24 k 2021-11-20T08:51:09.709 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 94 k 2021-11-20T08:51:09.709 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:51:09.715 INFO:teuthology.orchestra.run.smithi093.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 24 kB 00:00 2021-11-20T08:51:09.716 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:09.716 INFO:teuthology.orchestra.run.smithi093.stdout:Total 1.9 MB/s | 24 kB 00:00 2021-11-20T08:51:09.717 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:51:09.720 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:51:09.721 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:51:09.722 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:51:09.725 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:51:09.776 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:51:09.955 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:10.181 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:10.480 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:10.480 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:10.481 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:51:10.481 INFO:teuthology.orchestra.run.smithi093.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:10.481 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:10.481 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:10.605 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install librados2 2021-11-20T08:51:10.709 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:10.710 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:10.710 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:10.710 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:10.711 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:51:10.711 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T08:51:10.712 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:10.712 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:51:10.712 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:10.712 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-11-20T08:51:10.712 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:10.713 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 24 k 2021-11-20T08:51:10.713 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 94 k 2021-11-20T08:51:10.713 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:51:10.722 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 24 kB 00:00 2021-11-20T08:51:10.723 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:10.723 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.9 MB/s | 24 kB 00:00 2021-11-20T08:51:10.724 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:51:10.733 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:51:10.733 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:51:10.734 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:51:10.734 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:51:10.787 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:51:10.932 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:11.141 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:01:59 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:11.172 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:11.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:11.457 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:11.457 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:51:11.458 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:11.458 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:11.458 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:11.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2021-11-20T08:51:12.101 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:00 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:12.127 INFO:teuthology.orchestra.run.smithi093.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:12.418 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:12.419 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:51:12.419 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:12.749 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install librbd1 2021-11-20T08:51:13.140 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:13.296 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:01 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:13.424 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:13.425 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:51:13.425 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:13.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2021-11-20T08:51:14.032 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:02 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:14.293 INFO:teuthology.orchestra.run.smithi093.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:14.581 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:14.582 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:51:14.582 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:14.672 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install python3-rados 2021-11-20T08:51:15.214 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:03 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:15.621 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:15.914 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:15.915 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:51:15.915 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:16.218 INFO:teuthology.orchestra.run.smithi093.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:16.506 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:16.507 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:51:16.507 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:16.593 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install python3-rgw 2021-11-20T08:51:16.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2021-11-20T08:51:17.130 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:05 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:17.145 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:05 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:18.159 INFO:teuthology.orchestra.run.smithi093.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:18.168 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:18.450 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:18.450 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:51:18.451 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:18.457 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:18.459 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:51:18.460 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:18.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2021-11-20T08:51:18.704 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install python3-cephfs 2021-11-20T08:51:19.062 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:07 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:19.272 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:07 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:20.043 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:20.317 INFO:teuthology.orchestra.run.smithi093.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:20.331 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:20.332 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:51:20.333 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:20.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2021-11-20T08:51:20.609 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:51:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:20.697 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install python3-rbd 2021-11-20T08:51:20.939 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:08 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:21.232 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:09 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:21.993 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:22.259 INFO:teuthology.orchestra.run.smithi093.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:22.285 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:22.286 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:51:22.287 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:22.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2021-11-20T08:51:22.547 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:22.548 INFO:teuthology.orchestra.run.smithi093.stdout:Nothing to do. 2021-11-20T08:51:22.548 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:22.632 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install rbd-fuse 2021-11-20T08:51:22.888 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:10 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:23.168 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:11 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:23.917 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T08:51:24.205 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:24.206 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2021-11-20T08:51:24.207 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:24.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2021-11-20T08:51:24.468 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:24.469 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:24.470 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:24.470 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:24.471 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:51:24.471 INFO:teuthology.orchestra.run.smithi093.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T08:51:24.471 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:24.472 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:51:24.472 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:24.474 INFO:teuthology.orchestra.run.smithi093.stdout:Install 1 Package 2021-11-20T08:51:24.474 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:24.475 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 89 k 2021-11-20T08:51:24.475 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 298 k 2021-11-20T08:51:24.475 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:51:24.482 INFO:teuthology.orchestra.run.smithi093.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 6.8 MB/s | 89 kB 00:00 2021-11-20T08:51:24.482 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:24.483 INFO:teuthology.orchestra.run.smithi093.stdout:Total 6.1 MB/s | 89 kB 00:00 2021-11-20T08:51:24.483 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:51:24.495 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:51:24.496 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:51:24.546 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:51:24.549 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:51:24.805 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:51:24.808 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:12 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:25.120 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:25.367 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:25.695 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:25.696 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:25.696 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:51:25.696 INFO:teuthology.orchestra.run.smithi093.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:25.696 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:25.697 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:25.871 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install rbd-mirror 2021-11-20T08:51:26.106 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:26.108 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:26.108 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:26.108 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:26.109 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:51:26.110 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T08:51:26.110 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:26.111 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:51:26.111 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:26.112 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-11-20T08:51:26.112 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:26.112 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 89 k 2021-11-20T08:51:26.113 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 298 k 2021-11-20T08:51:26.113 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:51:26.120 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.7 MB/s | 89 kB 00:00 2021-11-20T08:51:26.121 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:26.121 INFO:teuthology.orchestra.run.smithi102.stdout:Total 6.9 MB/s | 89 kB 00:00 2021-11-20T08:51:26.121 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:51:26.133 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:51:26.133 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:51:26.182 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:51:26.186 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:51:26.288 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:51:26.415 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:14 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:26.447 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:26.715 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:27.046 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:27.046 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:27.046 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:51:27.047 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:27.047 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:27.047 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:27.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2021-11-20T08:51:27.693 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:27.694 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:27.695 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:27.695 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:27.695 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:51:27.696 INFO:teuthology.orchestra.run.smithi093.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T08:51:27.696 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:27.696 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:51:27.696 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:27.696 INFO:teuthology.orchestra.run.smithi093.stdout:Install 1 Package 2021-11-20T08:51:27.697 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:27.697 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 3.7 M 2021-11-20T08:51:27.697 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 18 M 2021-11-20T08:51:27.697 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:51:27.741 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:15 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:27.749 INFO:teuthology.orchestra.run.smithi093.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 69 MB/s | 3.7 MB 00:00 2021-11-20T08:51:27.750 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:27.750 INFO:teuthology.orchestra.run.smithi093.stdout:Total 67 MB/s | 3.7 MB 00:00 2021-11-20T08:51:27.751 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:51:27.765 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:51:27.765 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:51:27.829 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:51:27.832 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:51:28.294 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:51:28.512 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:28.909 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:29.010 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:29.011 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:29.012 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:29.012 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:29.012 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:51:29.012 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T08:51:29.013 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:29.013 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:51:29.013 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:29.013 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-11-20T08:51:29.014 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:29.014 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.7 M 2021-11-20T08:51:29.014 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 18 M 2021-11-20T08:51:29.014 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:51:29.067 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 69 MB/s | 3.7 MB 00:00 2021-11-20T08:51:29.067 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:29.068 INFO:teuthology.orchestra.run.smithi102.stdout:Total 67 MB/s | 3.7 MB 00:00 2021-11-20T08:51:29.068 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:51:29.082 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:51:29.082 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:51:29.146 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:51:29.150 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:51:29.244 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:29.244 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:29.244 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:51:29.245 INFO:teuthology.orchestra.run.smithi093.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:29.245 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:29.245 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:29.407 DEBUG:teuthology.orchestra.run.smithi093:> sudo yum -y install rbd-nbd 2021-11-20T08:51:29.614 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:51:29.823 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:29.955 INFO:teuthology.orchestra.run.smithi093.stdout:Last metadata expiration check: 0:02:17 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:30.201 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:30.519 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:30.519 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:30.519 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:51:30.520 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:30.520 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:30.520 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:30.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2021-11-20T08:51:31.239 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:19 ago on Sat 20 Nov 2021 08:49:12 AM UTC. 2021-11-20T08:51:31.246 INFO:teuthology.orchestra.run.smithi093.stdout:Dependencies resolved. 2021-11-20T08:51:31.247 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:31.248 INFO:teuthology.orchestra.run.smithi093.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:31.248 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:31.248 INFO:teuthology.orchestra.run.smithi093.stdout:Installing: 2021-11-20T08:51:31.248 INFO:teuthology.orchestra.run.smithi093.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T08:51:31.249 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:31.249 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction Summary 2021-11-20T08:51:31.249 INFO:teuthology.orchestra.run.smithi093.stdout:================================================================================ 2021-11-20T08:51:31.249 INFO:teuthology.orchestra.run.smithi093.stdout:Install 1 Package 2021-11-20T08:51:31.250 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:31.250 INFO:teuthology.orchestra.run.smithi093.stdout:Total download size: 173 k 2021-11-20T08:51:31.250 INFO:teuthology.orchestra.run.smithi093.stdout:Installed size: 649 k 2021-11-20T08:51:31.250 INFO:teuthology.orchestra.run.smithi093.stdout:Downloading Packages: 2021-11-20T08:51:31.262 INFO:teuthology.orchestra.run.smithi093.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 13 MB/s | 173 kB 00:00 2021-11-20T08:51:31.263 INFO:teuthology.orchestra.run.smithi093.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:31.263 INFO:teuthology.orchestra.run.smithi093.stdout:Total 12 MB/s | 173 kB 00:00 2021-11-20T08:51:31.264 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction check 2021-11-20T08:51:31.275 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction check succeeded. 2021-11-20T08:51:31.276 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction test 2021-11-20T08:51:31.325 INFO:teuthology.orchestra.run.smithi093.stdout:Transaction test succeeded. 2021-11-20T08:51:31.329 INFO:teuthology.orchestra.run.smithi093.stdout:Running transaction 2021-11-20T08:51:31.453 INFO:teuthology.orchestra.run.smithi093.stdout: Preparing : 1/1 2021-11-20T08:51:31.644 INFO:teuthology.orchestra.run.smithi093.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:31.902 INFO:teuthology.orchestra.run.smithi093.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:32.245 INFO:teuthology.orchestra.run.smithi093.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:32.245 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:32.246 INFO:teuthology.orchestra.run.smithi093.stdout:Installed: 2021-11-20T08:51:32.246 INFO:teuthology.orchestra.run.smithi093.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:32.246 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:51:32.246 INFO:teuthology.orchestra.run.smithi093.stdout:Complete! 2021-11-20T08:51:32.447 DEBUG:teuthology.parallel:result is None 2021-11-20T08:51:32.541 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2021-11-20T08:51:32.543 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:32.543 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2021-11-20T08:51:32.543 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:32.544 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2021-11-20T08:51:32.544 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T08:51:32.544 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:32.544 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2021-11-20T08:51:32.545 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2021-11-20T08:51:32.545 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2021-11-20T08:51:32.545 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:32.545 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 173 k 2021-11-20T08:51:32.547 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 649 k 2021-11-20T08:51:32.548 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2021-11-20T08:51:32.556 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 13 MB/s | 173 kB 00:00 2021-11-20T08:51:32.557 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2021-11-20T08:51:32.558 INFO:teuthology.orchestra.run.smithi102.stdout:Total 12 MB/s | 173 kB 00:00 2021-11-20T08:51:32.559 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2021-11-20T08:51:32.571 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2021-11-20T08:51:32.571 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2021-11-20T08:51:32.622 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2021-11-20T08:51:32.626 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2021-11-20T08:51:32.740 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2021-11-20T08:51:32.921 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:33.181 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:33.758 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T08:51:33.758 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:33.759 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2021-11-20T08:51:33.759 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T08:51:33.759 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:51:33.759 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2021-11-20T08:51:33.952 DEBUG:teuthology.parallel:result is None 2021-11-20T08:51:33.953 DEBUG:teuthology.orchestra.run.smithi093:> sudo lsb_release -is 2021-11-20T08:51:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:CentOSStream 2021-11-20T08:51:33.996 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T08:51:33.997 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T08:51:33.998 INFO:teuthology.packaging:ref: None 2021-11-20T08:51:33.998 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T08:51:33.998 INFO:teuthology.packaging:branch: None 2021-11-20T08:51:33.999 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T08:51:34.371 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:51:34.373 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:51:34.595 DEBUG:teuthology.orchestra.run.smithi093:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T08:51:34.621 INFO:teuthology.orchestra.run.smithi093.stdout:16.2.4-0.el8 2021-11-20T08:51:34.623 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T08:51:34.623 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T08:51:34.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo lsb_release -is 2021-11-20T08:51:34.664 INFO:teuthology.orchestra.run.smithi102.stdout:CentOSStream 2021-11-20T08:51:34.666 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T08:51:34.666 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T08:51:34.667 INFO:teuthology.packaging:ref: None 2021-11-20T08:51:34.667 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T08:51:34.667 INFO:teuthology.packaging:branch: None 2021-11-20T08:51:34.668 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T08:51:34.981 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:51:34.982 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T08:51:35.217 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T08:51:35.242 INFO:teuthology.orchestra.run.smithi102.stdout:16.2.4-0.el8 2021-11-20T08:51:35.243 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T08:51:35.243 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T08:51:35.244 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-11-20T08:51:35.244 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:51:35.245 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T08:51:35.276 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:51:35.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T08:51:35.312 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-11-20T08:51:35.313 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:51:35.314 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T08:51:35.345 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T08:51:35.414 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:51:35.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T08:51:35.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T08:51:35.518 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-11-20T08:51:35.519 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:51:35.519 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T08:51:35.548 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T08:51:35.617 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:51:35.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T08:51:35.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T08:51:35.712 INFO:teuthology.run_tasks:Running task cephadm... 2021-11-20T08:51:35.793 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T08:51:35.793 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2021-11-20T08:51:35.794 INFO:tasks.cephadm:Cluster fsid is 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:35.795 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-11-20T08:51:35.795 INFO:tasks.cephadm:No mon roles; fabricating mons 2021-11-20T08:51:35.795 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi093': '172.21.15.93', 'mon.smithi102': '172.21.15.102'} 2021-11-20T08:51:35.796 INFO:tasks.cephadm:Normalizing hostnames... 2021-11-20T08:51:35.796 DEBUG:teuthology.orchestra.run.smithi093:> sudo hostname $(hostname -s) 2021-11-20T08:51:35.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2021-11-20T08:51:35.852 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2021-11-20T08:51:35.853 DEBUG:teuthology.orchestra.run.smithi093:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T08:51:36.148 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 08:51 /home/ubuntu/cephtest/cephadm 2021-11-20T08:51:36.148 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T08:51:36.265 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 08:51 /home/ubuntu/cephtest/cephadm 2021-11-20T08:51:36.266 DEBUG:teuthology.orchestra.run.smithi093:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T08:51:36.285 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T08:51:36.345 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /etc/ceph 2021-11-20T08:51:36.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2021-11-20T08:51:36.402 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 777 /etc/ceph 2021-11-20T08:51:36.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2021-11-20T08:51:36.471 INFO:tasks.cephadm:Writing seed config... 2021-11-20T08:51:36.473 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2021-11-20T08:51:36.473 INFO:tasks.cephadm: override: [client] debug client = 20 2021-11-20T08:51:36.474 INFO:tasks.cephadm: override: [client] debug ms = 1 2021-11-20T08:51:36.474 INFO:tasks.cephadm: override: [client] rados mon op timeout = 15m 2021-11-20T08:51:36.474 INFO:tasks.cephadm: override: [client] rados osd op timeout = 15m 2021-11-20T08:51:36.474 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2021-11-20T08:51:36.475 INFO:tasks.cephadm: override: [mds] debug mds = 20 2021-11-20T08:51:36.475 INFO:tasks.cephadm: override: [mds] debug ms = 1 2021-11-20T08:51:36.475 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2021-11-20T08:51:36.476 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2021-11-20T08:51:36.476 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2021-11-20T08:51:36.476 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2021-11-20T08:51:36.478 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2021-11-20T08:51:36.478 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 15m 2021-11-20T08:51:36.479 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 15m 2021-11-20T08:51:36.479 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-11-20T08:51:36.479 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-11-20T08:51:36.480 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-11-20T08:51:36.480 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-11-20T08:51:36.480 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-11-20T08:51:36.481 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2021-11-20T08:51:36.481 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2021-11-20T08:51:36.481 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2021-11-20T08:51:36.481 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2021-11-20T08:51:36.482 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2021-11-20T08:51:36.482 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2021-11-20T08:51:36.482 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2021-11-20T08:51:36.483 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2021-11-20T08:51:36.483 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-11-20T08:51:36.483 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-11-20T08:51:36.484 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2021-11-20T08:51:36.484 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2021-11-20T08:51:36.484 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2021-11-20T08:51:36.485 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2021-11-20T08:51:36.485 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2021-11-20T08:51:36.485 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2021-11-20T08:51:36.486 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2021-11-20T08:51:36.487 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:51:36.487 DEBUG:teuthology.orchestra.run.smithi093:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-11-20T08:51:36.506 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 11458b34-49df-11ec-8c2c-001a4aab830c mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 15m rados osd op timeout = 15m [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 15m rados osd op timeout = 15m 2021-11-20T08:51:36.507 DEBUG:teuthology.orchestra.run.smithi093:mon.smithi093> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093.service 2021-11-20T08:51:36.549 INFO:tasks.cephadm:Bootstrapping... 2021-11-20T08:51:36.549 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 11458b34-49df-11ec-8c2c-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.93 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-11-20T08:51:36.576 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:-- Logs begin at Sat 2021-11-20 08:41:08 UTC. -- 2021-11-20T08:51:36.790 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T08:51:37.608 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: 3.4.1-dev 2021-11-20T08:51:37.682 INFO:teuthology.orchestra.run.smithi093.stderr:Verifying podman|docker is present... 2021-11-20T08:51:37.682 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T08:51:37.723 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: 3.4.1-dev 2021-11-20T08:51:37.827 INFO:teuthology.orchestra.run.smithi093.stderr:Verifying lvm2 is present... 2021-11-20T08:51:37.827 INFO:teuthology.orchestra.run.smithi093.stderr:Verifying time synchronization is in place... 2021-11-20T08:51:37.827 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T08:51:37.834 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T08:51:37.835 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active chrony.service 2021-11-20T08:51:37.840 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: inactive 2021-11-20T08:51:37.841 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T08:51:37.849 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: enabled 2021-11-20T08:51:37.849 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T08:51:37.851 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: active 2021-11-20T08:51:37.853 INFO:teuthology.orchestra.run.smithi093.stderr:Unit chronyd.service is enabled and running 2021-11-20T08:51:37.853 INFO:teuthology.orchestra.run.smithi093.stderr:Repeating the final host check... 2021-11-20T08:51:37.853 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T08:51:37.889 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: 3.4.1-dev 2021-11-20T08:51:37.993 INFO:teuthology.orchestra.run.smithi093.stderr:podman|docker (/bin/podman) is present 2021-11-20T08:51:37.993 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl is present 2021-11-20T08:51:37.995 INFO:teuthology.orchestra.run.smithi093.stderr:lvcreate is present 2021-11-20T08:51:37.995 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T08:51:38.007 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T08:51:38.008 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active chrony.service 2021-11-20T08:51:38.008 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: inactive 2021-11-20T08:51:38.008 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T08:51:38.009 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: enabled 2021-11-20T08:51:38.009 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T08:51:38.015 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: active 2021-11-20T08:51:38.015 INFO:teuthology.orchestra.run.smithi093.stderr:Unit chronyd.service is enabled and running 2021-11-20T08:51:38.016 INFO:teuthology.orchestra.run.smithi093.stderr:Host looks OK 2021-11-20T08:51:38.016 INFO:teuthology.orchestra.run.smithi093.stderr:Cluster fsid: 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:38.016 INFO:teuthology.orchestra.run.smithi093.stderr:Acquiring lock 140552188062408 on /run/cephadm/11458b34-49df-11ec-8c2c-001a4aab830c.lock 2021-11-20T08:51:38.017 INFO:teuthology.orchestra.run.smithi093.stderr:Lock 140552188062408 acquired on /run/cephadm/11458b34-49df-11ec-8c2c-001a4aab830c.lock 2021-11-20T08:51:38.018 INFO:teuthology.orchestra.run.smithi093.stderr:Verifying IP 172.21.15.93 port 3300 ... 2021-11-20T08:51:38.018 INFO:teuthology.orchestra.run.smithi093.stderr:Verifying IP 172.21.15.93 port 6789 ... 2021-11-20T08:51:38.019 INFO:teuthology.orchestra.run.smithi093.stderr:Base mon IP is 172.21.15.93, final addrv is [v2:172.21.15.93:3300,v1:172.21.15.93:6789] 2021-11-20T08:51:38.019 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /sbin/ip route ls 2021-11-20T08:51:38.019 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2021-11-20T08:51:38.020 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.93 metric 100 2021-11-20T08:51:38.021 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /sbin/ip -6 route ls 2021-11-20T08:51:38.021 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-11-20T08:51:38.021 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2021-11-20T08:51:38.022 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2021-11-20T08:51:38.022 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /sbin/ip -6 addr ls 2021-11-20T08:51:38.023 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-11-20T08:51:38.024 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-11-20T08:51:38.024 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T08:51:38.024 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2021-11-20T08:51:38.024 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2021-11-20T08:51:38.025 INFO:teuthology.orchestra.run.smithi093.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T08:51:38.026 INFO:teuthology.orchestra.run.smithi093.stderr:Mon IP 172.21.15.93 is in CIDR network 172.21.0.0/20 2021-11-20T08:51:38.026 INFO:teuthology.orchestra.run.smithi093.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-11-20T08:51:38.026 INFO:teuthology.orchestra.run.smithi093.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2021-11-20T08:51:38.026 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2021-11-20T08:51:38.093 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2021-11-20T08:51:38.513 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Getting image source signatures 2021-11-20T08:51:38.521 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T08:51:38.522 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T08:51:38.628 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T08:51:38.679 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T08:51:50.594 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T08:51:50.644 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Writing manifest to image destination 2021-11-20T08:51:50.644 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: Storing signatures 2021-11-20T08:51:51.000 INFO:teuthology.orchestra.run.smithi093.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T08:51:51.097 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2021-11-20T08:51:51.682 INFO:teuthology.orchestra.run.smithi093.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T08:51:52.044 INFO:teuthology.orchestra.run.smithi093.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T08:51:52.045 INFO:teuthology.orchestra.run.smithi093.stderr:Extracting ceph user uid/gid from container image... 2021-11-20T08:51:52.045 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2021-11-20T08:51:52.406 INFO:teuthology.orchestra.run.smithi093.stderr:stat: 167 167 2021-11-20T08:51:52.784 INFO:teuthology.orchestra.run.smithi093.stderr:Creating initial keys... 2021-11-20T08:51:52.784 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T08:51:53.130 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-authtool: AQApt5hheU+dBxAA4QGlLo65sdctDwGJ6m1w0g== 2021-11-20T08:51:53.830 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T08:51:54.180 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-authtool: AQAqt5hh4ceVChAAN5a42Jq1AZ8+Qqy0k/bBGA== 2021-11-20T08:51:54.480 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T08:51:54.847 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-authtool: AQAqt5hh9UQiMhAAlgZp/NLOu8W8Zsf/hZpuMQ== 2021-11-20T08:51:55.226 INFO:teuthology.orchestra.run.smithi093.stderr:Creating initial monmap... 2021-11-20T08:51:55.227 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpwim03fmk:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 11458b34-49df-11ec-8c2c-001a4aab830c --addv smithi093 [v2:172.21.15.93:3300,v1:172.21.15.93:6789] /tmp/monmap 2021-11-20T08:51:55.547 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T08:51:55.548 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:55.548 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T08:51:55.865 INFO:teuthology.orchestra.run.smithi093.stderr:monmaptool for smithi093 [v2:172.21.15.93:3300,v1:172.21.15.93:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T08:51:55.865 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/monmaptool: set fsid to 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:55.866 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T08:51:55.866 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:51:55.866 INFO:teuthology.orchestra.run.smithi093.stderr:Creating mon... 2021-11-20T08:51:55.867 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093:/var/lib/ceph/mon/ceph-smithi093:z -v /tmp/ceph-tmpir5iclxa:/tmp/keyring:z -v /tmp/ceph-tmpwim03fmk:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi093 --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-11-20T08:51:56.250 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.214+0000 7f03302f1700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T08:51:56.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.215+0000 7f03302f1700 1 imported monmap: 2021-11-20T08:51:56.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: epoch 0 2021-11-20T08:51:56.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: fsid 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:56.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: last_changed 2021-11-20T08:51:55.545573+0000 2021-11-20T08:51:56.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: created 2021-11-20T08:51:55.545573+0000 2021-11-20T08:51:56.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-11-20T08:51:56.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-11-20T08:51:56.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] mon.smithi093 2021-11-20T08:51:56.253 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.253 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.215+0000 7f03302f1700 0 /usr/bin/ceph-mon: set fsid to 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:56.253 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T08:51:56.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T08:51:56.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Compile date May 13 2021 2021-11-20T08:51:56.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: DB SUMMARY 2021-11-20T08:51:56.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi093/store.db dir, Total Num: 0, files: 2021-11-20T08:51:56.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi093/store.db: 2021-11-20T08:51:56.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T08:51:56.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.create_if_missing: 1 2021-11-20T08:51:56.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T08:51:56.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.env: 0x55b9afcc41c0 2021-11-20T08:51:56.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.fs: Posix File System 2021-11-20T08:51:56.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.info_log: 0x55b9b20c2ec0 2021-11-20T08:51:56.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T08:51:56.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.statistics: (nil) 2021-11-20T08:51:56.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.use_fsync: 0 2021-11-20T08:51:56.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T08:51:56.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T08:51:56.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T08:51:56.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T08:51:56.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T08:51:56.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T08:51:56.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T08:51:56.260 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T08:51:56.260 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T08:51:56.260 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T08:51:56.260 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T08:51:56.261 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.db_log_dir: 2021-11-20T08:51:56.261 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi093/store.db 2021-11-20T08:51:56.261 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T08:51:56.261 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T08:51:56.261 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T08:51:56.262 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T08:51:56.262 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T08:51:56.262 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T08:51:56.262 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T08:51:56.263 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T08:51:56.263 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T08:51:56.263 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T08:51:56.263 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.write_buffer_manager: 0x55b9b20c9530 2021-11-20T08:51:56.264 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T08:51:56.264 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T08:51:56.264 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T08:51:56.264 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T08:51:56.264 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T08:51:56.265 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T08:51:56.265 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T08:51:56.265 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T08:51:56.265 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T08:51:56.265 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.unordered_write: 0 2021-11-20T08:51:56.266 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T08:51:56.266 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T08:51:56.266 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T08:51:56.266 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T08:51:56.267 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.row_cache: None 2021-11-20T08:51:56.267 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.wal_filter: None 2021-11-20T08:51:56.267 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T08:51:56.267 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T08:51:56.267 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T08:51:56.268 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T08:51:56.268 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T08:51:56.268 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T08:51:56.268 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T08:51:56.269 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T08:51:56.269 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T08:51:56.269 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T08:51:56.269 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T08:51:56.270 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T08:51:56.270 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T08:51:56.270 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T08:51:56.270 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T08:51:56.270 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T08:51:56.271 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T08:51:56.271 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T08:51:56.271 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T08:51:56.271 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T08:51:56.272 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T08:51:56.272 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.max_open_files: -1 2021-11-20T08:51:56.272 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T08:51:56.272 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T08:51:56.272 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T08:51:56.273 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T08:51:56.273 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Compression algorithms supported: 2021-11-20T08:51:56.273 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T08:51:56.273 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kZSTD supported: 0 2021-11-20T08:51:56.274 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T08:51:56.274 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T08:51:56.274 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T08:51:56.274 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T08:51:56.275 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T08:51:56.275 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T08:51:56.275 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T08:51:56.275 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.216+0000 7f03302f1700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-11-20T08:51:56.276 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.276 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi093/store.db/MANIFEST-000001 2021-11-20T08:51:56.276 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.276 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T08:51:56.276 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.277 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T08:51:56.277 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.merge_operator: 2021-11-20T08:51:56.277 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_filter: None 2021-11-20T08:51:56.277 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T08:51:56.278 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T08:51:56.278 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T08:51:56.278 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b9b1fda120) 2021-11-20T08:51:56.278 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-11-20T08:51:56.279 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T08:51:56.279 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T08:51:56.279 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-11-20T08:51:56.279 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: index_type: 0 2021-11-20T08:51:56.280 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-11-20T08:51:56.280 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-11-20T08:51:56.280 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-11-20T08:51:56.280 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-11-20T08:51:56.281 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: checksum: 1 2021-11-20T08:51:56.281 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-11-20T08:51:56.281 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_cache: 0x55b9b2010d10 2021-11-20T08:51:56.282 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-11-20T08:51:56.282 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-11-20T08:51:56.282 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-11-20T08:51:56.282 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-11-20T08:51:56.282 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-11-20T08:51:56.283 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-11-20T08:51:56.283 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-11-20T08:51:56.283 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-11-20T08:51:56.283 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-11-20T08:51:56.284 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-11-20T08:51:56.284 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-11-20T08:51:56.284 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-11-20T08:51:56.284 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-11-20T08:51:56.284 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-11-20T08:51:56.285 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-11-20T08:51:56.285 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T08:51:56.285 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-11-20T08:51:56.285 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-11-20T08:51:56.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-11-20T08:51:56.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: format_version: 2 2021-11-20T08:51:56.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-11-20T08:51:56.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: block_align: 0 2021-11-20T08:51:56.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T08:51:56.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T08:51:56.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compression: NoCompression 2021-11-20T08:51:56.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T08:51:56.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T08:51:56.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T08:51:56.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.num_levels: 7 2021-11-20T08:51:56.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T08:51:56.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T08:51:56.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T08:51:56.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T08:51:56.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T08:51:56.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T08:51:56.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T08:51:56.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T08:51:56.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T08:51:56.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T08:51:56.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T08:51:56.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T08:51:56.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T08:51:56.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T08:51:56.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T08:51:56.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T08:51:56.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T08:51:56.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T08:51:56.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T08:51:56.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T08:51:56.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T08:51:56.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T08:51:56.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T08:51:56.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T08:51:56.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T08:51:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T08:51:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T08:51:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T08:51:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T08:51:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T08:51:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T08:51:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T08:51:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T08:51:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T08:51:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T08:51:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T08:51:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T08:51:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T08:51:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T08:51:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T08:51:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T08:51:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T08:51:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T08:51:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T08:51:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T08:51:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T08:51:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T08:51:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T08:51:56.299 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T08:51:56.299 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T08:51:56.299 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T08:51:56.299 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T08:51:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T08:51:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T08:51:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T08:51:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T08:51:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T08:51:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T08:51:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T08:51:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.ttl: 2592000 2021-11-20T08:51:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T08:51:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi093/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T08:51:56.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.219+0000 7f03302f1700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T08:51:56.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.220+0000 7f03302f1700 4 rocksdb: DB pointer 0x55b9b20d5800 2021-11-20T08:51:56.303 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.220+0000 7f031918a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T08:51:56.303 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.220+0000 7f031918a700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T08:51:56.303 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-11-20T08:51:56.303 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:51:56.304 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T08:51:56.304 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T08:51:56.304 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T08:51:56.304 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T08:51:56.305 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T08:51:56.305 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T08:51:56.305 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.305 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T08:51:56.305 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:51:56.306 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:51:56.306 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T08:51:56.306 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T08:51:56.306 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.306 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T08:51:56.307 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:51:56.307 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:51:56.307 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:51:56.307 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T08:51:56.308 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T08:51:56.308 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T08:51:56.308 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T08:51:56.308 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T08:51:56.308 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:51:56.309 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:51:56.309 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T08:51:56.309 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.309 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T08:51:56.310 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.310 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T08:51:56.310 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:51:56.310 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:51:56.310 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T08:51:56.311 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T08:51:56.311 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.311 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T08:51:56.311 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:51:56.312 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:51:56.312 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:51:56.312 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T08:51:56.312 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T08:51:56.312 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T08:51:56.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T08:51:56.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T08:51:56.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:51:56.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:51:56.314 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T08:51:56.314 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.314 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T08:51:56.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: 2021-11-20T08:51:56.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.221+0000 7f03302f1700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-11-20T08:51:56.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.221+0000 7f03302f1700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-11-20T08:51:56.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph-mon: debug 2021-11-20T08:51:56.222+0000 7f03302f1700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi093 for mon.smithi093 2021-11-20T08:51:56.516 INFO:teuthology.orchestra.run.smithi093.stderr:create mon.smithi093 on 2021-11-20T08:51:56.517 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:56.522 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl disable ceph.target 2021-11-20T08:51:56.528 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-11-20T08:51:56.610 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl enable ceph.target 2021-11-20T08:51:56.617 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-11-20T08:51:56.697 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl start ceph.target 2021-11-20T08:51:56.704 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl enable ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target 2021-11-20T08:51:56.710 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target. 2021-11-20T08:51:56.710 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target. 2021-11-20T08:51:56.789 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl start ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target 2021-11-20T08:51:56.796 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl daemon-reload 2021-11-20T08:51:56.879 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093 2021-11-20T08:51:56.906 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl reset-failed ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093 2021-11-20T08:51:56.912 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Failed to reset failed state of unit ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093.service: Unit ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093.service not loaded. 2021-11-20T08:51:56.913 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl enable ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093 2021-11-20T08:51:56.918 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Created symlink /etc/systemd/system/ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target.wants/ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093.service → /etc/systemd/system/ceph-11458b34-49df-11ec-8c2c-001a4aab830c@.service. 2021-11-20T08:51:57.000 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl start ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093 2021-11-20T08:51:57.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:57 smithi093 systemd[1]: Starting Ceph mon.smithi093 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T08:51:57.474 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:57 smithi093 podman[32782]: 2021-11-20T08:51:57.474 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:57 smithi093 bash[32706]: fe2e49c38f272c88af65d4a955adaf16544efa1d8777a302beb2c347b587c612 2021-11-20T08:51:57.527 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T08:51:57.536 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: disabled 2021-11-20T08:51:57.536 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T08:51:57.542 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: inactive 2021-11-20T08:51:57.543 INFO:teuthology.orchestra.run.smithi093.stderr:firewalld.service is not enabled 2021-11-20T08:51:57.543 INFO:teuthology.orchestra.run.smithi093.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T08:51:57.544 INFO:teuthology.orchestra.run.smithi093.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T08:51:57.545 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for mon to start... 2021-11-20T08:51:57.545 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for mon... 2021-11-20T08:51:57.545 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093:/var/lib/ceph/mon/ceph-smithi093:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2021-11-20T08:51:57.732 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:57 smithi093 conmon[32822]: cluster 2021-11-20T08:51:57.464556+0000 mon.smithi093 (mon.0) 0 : [INF] mkfs 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:57.733 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:57 smithi093 conmon[32822]: cluster 2021-11-20T08:51 2021-11-20T08:51:57.733 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:57 smithi093 conmon[32822]: :57.450428+0000 mon.smithi093 (mon.0) 1 : cluster [INF] mon.smithi093 is new leader, mons smithi093 in quorum (ranks 0) 2021-11-20T08:51:57.734 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:57 smithi093 systemd[1]: Started Ceph mon.smithi093 for 11458b34-49df-11ec-8c2c-001a4aab830c. 2021-11-20T08:51:57.979 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.972+0000 7fd38450f700 1 Processor -- start 2021-11-20T08:51:57.979 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.972+0000 7fd38450f700 1 -- start start 2021-11-20T08:51:57.980 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.973+0000 7fd38450f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c0faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:51:57.980 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.973+0000 7fd38450f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0fb3e0 con 0x7fd37c0faad0 2021-11-20T08:51:57.980 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.973+0000 7fd3822ab700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c0faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:51:57.981 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.973+0000 7fd3822ab700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c0faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42060/0 (socket says 172.21.15.93:42060) 2021-11-20T08:51:57.981 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.973+0000 7fd3822ab700 1 -- 172.21.15.93:0/3167924689 learned_addr learned my addr 172.21.15.93:0/3167924689 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:51:57.981 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.973+0000 7fd3822ab700 1 -- 172.21.15.93:0/3167924689 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c0fb520 con 0x7fd37c0faad0 2021-11-20T08:51:57.982 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd3822ab700 1 --2- 172.21.15.93:0/3167924689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c0faef0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd37800b870 tx=0x7fd378006f70).ready entity=mon.0 client_cookie=4506013e8660a4f7 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:51:57.982 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd3812a9700 1 -- 172.21.15.93:0/3167924689 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd37800da20 con 0x7fd37c0faad0 2021-11-20T08:51:57.983 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd3812a9700 1 -- 172.21.15.93:0/3167924689 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37800db80 con 0x7fd37c0faad0 2021-11-20T08:51:57.983 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd3812a9700 1 -- 172.21.15.93:0/3167924689 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd37800de50 con 0x7fd37c0faad0 2021-11-20T08:51:57.983 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd38450f700 1 -- 172.21.15.93:0/3167924689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 msgr2=0x7fd37c0faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:51:57.984 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd38450f700 1 --2- 172.21.15.93:0/3167924689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c0faef0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd37800b870 tx=0x7fd378006f70).stop 2021-11-20T08:51:57.984 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd38450f700 1 -- 172.21.15.93:0/3167924689 shutdown_connections 2021-11-20T08:51:57.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd38450f700 1 --2- 172.21.15.93:0/3167924689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c0faef0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:51:57.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd38450f700 1 -- 172.21.15.93:0/3167924689 >> 172.21.15.93:0/3167924689 conn(0x7fd37c0f6310 msgr2=0x7fd37c0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:51:57.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd38450f700 1 -- 172.21.15.93:0/3167924689 shutdown_connections 2021-11-20T08:51:57.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.974+0000 7fd38450f700 1 -- 172.21.15.93:0/3167924689 wait complete. 2021-11-20T08:51:57.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.975+0000 7fd38450f700 1 Processor -- start 2021-11-20T08:51:57.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.975+0000 7fd38450f700 1 -- start start 2021-11-20T08:51:57.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.975+0000 7fd38450f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c107f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:51:57.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.975+0000 7fd38450f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c108450 con 0x7fd37c0faad0 2021-11-20T08:51:57.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd3822ab700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:51:57.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd3822ab700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c107f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42062/0 (socket says 172.21.15.93:42062) 2021-11-20T08:51:57.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd3822ab700 1 -- 172.21.15.93:0/1691816623 learned_addr learned my addr 172.21.15.93:0/1691816623 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:51:57.988 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd3822ab700 1 -- 172.21.15.93:0/1691816623 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37800b480 con 0x7fd37c0faad0 2021-11-20T08:51:57.988 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd3822ab700 1 --2- 172.21.15.93:0/1691816623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c107f60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd378006c70 tx=0x7fd378006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:51:57.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd36affd700 1 -- 172.21.15.93:0/1691816623 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd37801b110 con 0x7fd37c0faad0 2021-11-20T08:51:57.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd36affd700 1 -- 172.21.15.93:0/1691816623 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37801b270 con 0x7fd37c0faad0 2021-11-20T08:51:57.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37c108650 con 0x7fd37c0faad0 2021-11-20T08:51:57.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd36affd700 1 -- 172.21.15.93:0/1691816623 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3780105d0 con 0x7fd37c0faad0 2021-11-20T08:51:57.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37c108a60 con 0x7fd37c0faad0 2021-11-20T08:51:57.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd36affd700 1 -- 172.21.15.93:0/1691816623 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd37802c070 con 0x7fd37c0faad0 2021-11-20T08:51:57.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.976+0000 7fd36affd700 1 -- 172.21.15.93:0/1691816623 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd3780197c0 con 0x7fd37c0faad0 2021-11-20T08:51:57.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.977+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37c045ad0 con 0x7fd37c0faad0 2021-11-20T08:51:57.991 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:57.979+0000 7fd36affd700 1 -- 172.21.15.93:0/1691816623 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd3780449c0 con 0x7fd37c0faad0 2021-11-20T08:51:58.050 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.045+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd37c0fbe10 con 0x7fd37c0faad0 2021-11-20T08:51:58.050 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.045+0000 7fd36affd700 1 -- 172.21.15.93:0/1691816623 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fd378033070 con 0x7fd37c0faad0 2021-11-20T08:51:58.051 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: cluster: 2021-11-20T08:51:58.051 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: id: 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:58.051 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-11-20T08:51:58.051 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: services: 2021-11-20T08:51:58.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi093 (age 0.587338s) 2021-11-20T08:51:58.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mgr: no daemons active 2021-11-20T08:51:58.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-11-20T08:51:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: data: 2021-11-20T08:51:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-11-20T08:51:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-11-20T08:51:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-11-20T08:51:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: pgs: 2021-11-20T08:51:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.046+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 msgr2=0x7fd37c107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:51:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.046+0000 7fd38450f700 1 --2- 172.21.15.93:0/1691816623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c107f60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd378006c70 tx=0x7fd378006f70).stop 2021-11-20T08:51:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.046+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 shutdown_connections 2021-11-20T08:51:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.046+0000 7fd38450f700 1 --2- 172.21.15.93:0/1691816623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd37c0faad0 0x7fd37c107f60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:51:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.046+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 >> 172.21.15.93:0/1691816623 conn(0x7fd37c0f6310 msgr2=0x7fd37c0f8570 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:51:58.056 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.047+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 shutdown_connections 2021-11-20T08:51:58.056 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.047+0000 7fd38450f700 1 -- 172.21.15.93:0/1691816623 wait complete. 2021-11-20T08:51:58.384 INFO:teuthology.orchestra.run.smithi093.stderr:mon is available 2021-11-20T08:51:58.385 INFO:teuthology.orchestra.run.smithi093.stderr:Assimilating anything we can from ceph.conf... 2021-11-20T08:51:58.385 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z -v /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093:/var/lib/ceph/mon/ceph-smithi093:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi093/config 2021-11-20T08:51:58.632 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:58 smithi093 conmon[32822]: cluster 2021-11-20T08:51:57.459220+0000 mon.smithi093 (mon.0) 2 : cluster [INF] mon.smithi093 is new leader, mons smithi093 in quorum (ranks 0) 2021-11-20T08:51:58.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:58 smithi093 conmon[32822]: cluster 2021-11-20T08:51:57.461653+0000 mon.smithi093 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi093=[v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0]} 2021-11-20T08:51:58.634 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:58 smithi093 conmon[32822]: cluster 2021-11-20T08:51:57.470176+0000 mon.smithi093 (mon.0) 4 : cluster [DBG] fsmap 2021-11-20T08:51:58.635 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:58 smithi093 conmon[32822]: cluster 2021-11-20T08:51:57.477103+0000 mon.smithi093 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T08:51:58.635 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:58 smithi093 conmon[32822]: cluster 2021-11-20T08:51:57.477491+0000 mon.smithi093 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T08:51:58.636 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:58 smithi093 conmon[32822]: audit 2021-11-20T08:51:58.046518+0000 mon.smithi093 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.93:0/1691816623' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-11-20T08:51:58.847 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.844+0000 7f17342a4700 1 Processor -- start 2021-11-20T08:51:58.847 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.844+0000 7f17342a4700 1 -- start start 2021-11-20T08:51:58.848 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.845+0000 7f17342a4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c0fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:51:58.848 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.845+0000 7f17342a4700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172c06ff70 con 0x7f172c06fb50 2021-11-20T08:51:58.848 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.845+0000 7f1732040700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c0fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:51:58.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.845+0000 7f1732040700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c0fe910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42064/0 (socket says 172.21.15.93:42064) 2021-11-20T08:51:58.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.845+0000 7f1732040700 1 -- 172.21.15.93:0/1673803869 learned_addr learned my addr 172.21.15.93:0/1673803869 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:51:58.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f1732040700 1 -- 172.21.15.93:0/1673803869 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172c0fee00 con 0x7f172c06fb50 2021-11-20T08:51:58.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f1732040700 1 --2- 172.21.15.93:0/1673803869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c0fe910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f172800b5f0 tx=0x7f1728006ee0).ready entity=mon.0 client_cookie=c90ff184c735fc87 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:51:58.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f173103e700 1 -- 172.21.15.93:0/1673803869 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f172800d600 con 0x7f172c06fb50 2021-11-20T08:51:58.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f173103e700 1 -- 172.21.15.93:0/1673803869 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172800d760 con 0x7f172c06fb50 2021-11-20T08:51:58.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f173103e700 1 -- 172.21.15.93:0/1673803869 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f172800da30 con 0x7f172c06fb50 2021-11-20T08:51:58.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f17342a4700 1 -- 172.21.15.93:0/1673803869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 msgr2=0x7f172c0fe910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:51:58.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f17342a4700 1 --2- 172.21.15.93:0/1673803869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c0fe910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f172800b5f0 tx=0x7f1728006ee0).stop 2021-11-20T08:51:58.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f17342a4700 1 -- 172.21.15.93:0/1673803869 shutdown_connections 2021-11-20T08:51:58.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f17342a4700 1 --2- 172.21.15.93:0/1673803869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c0fe910 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:51:58.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f17342a4700 1 -- 172.21.15.93:0/1673803869 >> 172.21.15.93:0/1673803869 conn(0x7f172c0f6310 msgr2=0x7f172c0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:51:58.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.846+0000 7f17342a4700 1 -- 172.21.15.93:0/1673803869 shutdown_connections 2021-11-20T08:51:58.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f17342a4700 1 -- 172.21.15.93:0/1673803869 wait complete. 2021-11-20T08:51:58.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f17342a4700 1 Processor -- start 2021-11-20T08:51:58.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f17342a4700 1 -- start start 2021-11-20T08:51:58.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f17342a4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c10a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:51:58.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f17342a4700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172c10a710 con 0x7f172c06fb50 2021-11-20T08:51:58.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f1732040700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c10a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:51:58.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f1732040700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c10a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42066/0 (socket says 172.21.15.93:42066) 2021-11-20T08:51:58.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.847+0000 7f1732040700 1 -- 172.21.15.93:0/2610969771 learned_addr learned my addr 172.21.15.93:0/2610969771 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:51:58.854 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f1732040700 1 -- 172.21.15.93:0/2610969771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172800b200 con 0x7f172c06fb50 2021-11-20T08:51:58.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f1732040700 1 --2- 172.21.15.93:0/2610969771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c10a220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1728006c70 tx=0x7f1728016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:51:58.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1728011b40 con 0x7f172c06fb50 2021-11-20T08:51:58.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1728011ca0 con 0x7f172c06fb50 2021-11-20T08:51:58.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f172801d430 con 0x7f172c06fb50 2021-11-20T08:51:58.856 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172c10a910 con 0x7f172c06fb50 2021-11-20T08:51:58.856 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172c10ad20 con 0x7f172c06fb50 2021-11-20T08:51:58.856 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f172800f070 con 0x7f172c06fb50 2021-11-20T08:51:58.856 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.848+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1728027460 con 0x7f172c06fb50 2021-11-20T08:51:58.856 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.849+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172c045ad0 con 0x7f172c06fb50 2021-11-20T08:51:58.857 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.852+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1728016070 con 0x7f172c06fb50 2021-11-20T08:51:58.914 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.916+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f172c0ff170 con 0x7f172c06fb50 2021-11-20T08:51:58.920 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.919+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f172801d660 con 0x7f172c06fb50 2021-11-20T08:51:58.921 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.919+0000 7f171effd700 1 -- 172.21.15.93:0/2610969771 <== mon.0 v2:172.21.15.93:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f1728036020 con 0x7f172c06fb50 2021-11-20T08:51:58.921 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.921 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: [global] 2021-11-20T08:51:58.921 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: fsid = 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:51:58.922 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.93:3300,v1:172.21.15.93:6789] 2021-11-20T08:51:58.922 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-11-20T08:51:58.922 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-11-20T08:51:58.922 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-11-20T08:51:58.922 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-11-20T08:51:58.923 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.923 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: [mgr] 2021-11-20T08:51:58.923 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-11-20T08:51:58.923 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: [osd] 2021-11-20T08:51:58.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2021-11-20T08:51:58.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2021-11-20T08:51:58.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2021-11-20T08:51:58.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-11-20T08:51:58.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2021-11-20T08:51:58.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-11-20T08:51:58.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.920+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 msgr2=0x7f172c10a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:51:58.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.920+0000 7f17342a4700 1 --2- 172.21.15.93:0/2610969771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c10a220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1728006c70 tx=0x7f1728016000).stop 2021-11-20T08:51:58.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.920+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 shutdown_connections 2021-11-20T08:51:58.926 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.920+0000 7f17342a4700 1 --2- 172.21.15.93:0/2610969771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f172c06fb50 0x7f172c10a220 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:51:58.926 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.920+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 >> 172.21.15.93:0/2610969771 conn(0x7f172c0f6310 msgr2=0x7f172c0f8070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:51:58.926 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.920+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 shutdown_connections 2021-11-20T08:51:58.926 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:58.920+0000 7f17342a4700 1 -- 172.21.15.93:0/2610969771 wait complete. 2021-11-20T08:51:59.322 INFO:teuthology.orchestra.run.smithi093.stderr:Generating new minimal ceph.conf... 2021-11-20T08:51:59.322 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z -v /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093:/var/lib/ceph/mon/ceph-smithi093:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi093/config 2021-11-20T08:51:59.527 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:59 smithi093 conmon[32822]: audit 2021-11-20T08:51:58.917036+0000 mon.smithi093 (mon.0) 8 : audit [INF] from='client.? 172.21.15.93:0/2610969771' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-20T08:51:59.527 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:59 smithi093 conmon[32822]: audit 2021-11-20T08:51: 2021-11-20T08:51:59.527 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:51:59 smithi093 conmon[32822]: 58.920248+0000 mon.smithi093 (mon.0) 9 : audit [INF] from='client.? 172.21.15.93:0/2610969771' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-20T08:51:59.781 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.778+0000 7ff79f9ae700 1 Processor -- start 2021-11-20T08:51:59.781 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.778+0000 7ff79f9ae700 1 -- start start 2021-11-20T08:51:59.782 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff79f9ae700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff7980fc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:51:59.782 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff79f9ae700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980fcad0 con 0x7ff7980fc1c0 2021-11-20T08:51:59.782 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff79d74a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff7980fc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:51:59.783 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff79d74a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff7980fc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42068/0 (socket says 172.21.15.93:42068) 2021-11-20T08:51:59.783 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff79d74a700 1 -- 172.21.15.93:0/1619629683 learned_addr learned my addr 172.21.15.93:0/1619629683 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:51:59.783 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff79d74a700 1 -- 172.21.15.93:0/1619629683 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7980fcc10 con 0x7ff7980fc1c0 2021-11-20T08:51:59.783 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff79d74a700 1 --2- 172.21.15.93:0/1619629683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff7980fc5e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff794009960 tx=0x7ff7940092d0).ready entity=mon.0 client_cookie=2490170f4d740023 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:51:59.783 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff787fff700 1 -- 172.21.15.93:0/1619629683 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff79400b9c0 con 0x7ff7980fc1c0 2021-11-20T08:51:59.784 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff787fff700 1 -- 172.21.15.93:0/1619629683 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff794011400 con 0x7ff7980fc1c0 2021-11-20T08:51:59.784 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.779+0000 7ff787fff700 1 -- 172.21.15.93:0/1619629683 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7940103a0 con 0x7ff7980fc1c0 2021-11-20T08:51:59.784 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.780+0000 7ff79f9ae700 1 -- 172.21.15.93:0/1619629683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 msgr2=0x7ff7980fc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:51:59.784 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.780+0000 7ff79f9ae700 1 --2- 172.21.15.93:0/1619629683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff7980fc5e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff794009960 tx=0x7ff7940092d0).stop 2021-11-20T08:51:59.785 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.780+0000 7ff79f9ae700 1 -- 172.21.15.93:0/1619629683 shutdown_connections 2021-11-20T08:51:59.785 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.780+0000 7ff79f9ae700 1 --2- 172.21.15.93:0/1619629683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff7980fc5e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:51:59.785 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.780+0000 7ff79f9ae700 1 -- 172.21.15.93:0/1619629683 >> 172.21.15.93:0/1619629683 conn(0x7ff7980f5a70 msgr2=0x7ff7980f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:51:59.785 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.780+0000 7ff79f9ae700 1 -- 172.21.15.93:0/1619629683 shutdown_connections 2021-11-20T08:51:59.785 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.780+0000 7ff79f9ae700 1 -- 172.21.15.93:0/1619629683 wait complete. 2021-11-20T08:51:59.786 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79f9ae700 1 Processor -- start 2021-11-20T08:51:59.786 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79f9ae700 1 -- start start 2021-11-20T08:51:59.786 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79f9ae700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff79810cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:51:59.786 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79f9ae700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79810d090 con 0x7ff7980fc1c0 2021-11-20T08:51:59.787 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79d74a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff79810cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:51:59.787 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79d74a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff79810cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42070/0 (socket says 172.21.15.93:42070) 2021-11-20T08:51:59.787 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79d74a700 1 -- 172.21.15.93:0/3659847312 learned_addr learned my addr 172.21.15.93:0/3659847312 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:51:59.787 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.781+0000 7ff79d74a700 1 -- 172.21.15.93:0/3659847312 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff794009570 con 0x7ff7980fc1c0 2021-11-20T08:51:59.787 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff79d74a700 1 --2- 172.21.15.93:0/3659847312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff79810cba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff794000f80 tx=0x7ff7940092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:51:59.788 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff785ffb700 1 -- 172.21.15.93:0/3659847312 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff794010820 con 0x7ff7980fc1c0 2021-11-20T08:51:59.788 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff785ffb700 1 -- 172.21.15.93:0/3659847312 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff794010df0 con 0x7ff7980fc1c0 2021-11-20T08:51:59.788 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff785ffb700 1 -- 172.21.15.93:0/3659847312 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff79401ae40 con 0x7ff7980fc1c0 2021-11-20T08:51:59.788 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79810d290 con 0x7ff7980fc1c0 2021-11-20T08:51:59.789 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79810d6a0 con 0x7ff7980fc1c0 2021-11-20T08:51:59.789 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff785ffb700 1 -- 172.21.15.93:0/3659847312 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff79400f070 con 0x7ff7980fc1c0 2021-11-20T08:51:59.789 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.782+0000 7ff785ffb700 1 -- 172.21.15.93:0/3659847312 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff79402c760 con 0x7ff7980fc1c0 2021-11-20T08:51:59.790 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.783+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff798045ad0 con 0x7ff7980fc1c0 2021-11-20T08:51:59.790 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.785+0000 7ff785ffb700 1 -- 172.21.15.93:0/3659847312 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff79401e030 con 0x7ff7980fc1c0 2021-11-20T08:51:59.848 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.849+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff798106980 con 0x7ff7980fc1c0 2021-11-20T08:51:59.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff785ffb700 1 -- 172.21.15.93:0/3659847312 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ff79401a430 con 0x7ff7980fc1c0 2021-11-20T08:51:59.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 msgr2=0x7ff79810cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:51:59.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff79f9ae700 1 --2- 172.21.15.93:0/3659847312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff79810cba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff794000f80 tx=0x7ff7940092d0).stop 2021-11-20T08:51:59.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 shutdown_connections 2021-11-20T08:51:59.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff79f9ae700 1 --2- 172.21.15.93:0/3659847312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7980fc1c0 0x7ff79810cba0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:51:59.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 >> 172.21.15.93:0/3659847312 conn(0x7ff7980f5a70 msgr2=0x7ff7980f7580 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:51:59.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 shutdown_connections 2021-11-20T08:51:59.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:51:59.850+0000 7ff79f9ae700 1 -- 172.21.15.93:0/3659847312 wait complete. 2021-11-20T08:52:00.258 INFO:teuthology.orchestra.run.smithi093.stderr:Restarting the monitor... 2021-11-20T08:52:00.259 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl restart ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093 2021-11-20T08:52:00.344 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:00 smithi093 systemd[1]: Stopping Ceph mon.smithi093 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T08:52:00.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:00 smithi093 conmon[32822]: debug 2021-11-20T08:52:00.345+0000 7fe0f6185700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi093 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T08:52:00.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:00 smithi093 conmon[32822]: debug 2021-11-20T08:52:00.345+0000 7fe0f6185700 -1 mon.smithi093@0(leader) e1 *** Got Signal Terminated *** 2021-11-20T08:52:01.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:00 smithi093 podman[33304]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-mon.smithi093 2021-11-20T08:52:01.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 systemd[1]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093.service: Succeeded. 2021-11-20T08:52:01.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 systemd[1]: Stopped Ceph mon.smithi093 for 11458b34-49df-11ec-8c2c-001a4aab830c. 2021-11-20T08:52:01.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 systemd[1]: Starting Ceph mon.smithi093 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T08:52:01.472 INFO:teuthology.orchestra.run.smithi093.stderr:Setting mon public_network to 172.21.0.0/20 2021-11-20T08:52:01.473 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2021-11-20T08:52:01.607 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 podman[33482]: 2021-11-20T08:52:01.608 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.439+0000 7f018fd56700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T08:52:01.608 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.439+0000 7f018fd56700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-20T08:52:01.608 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.439+0000 7f018fd56700 0 pidfile_write: ignore empty --pid-file 2021-11-20T08:52:01.609 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 0 load: jerasure load: lrc load: isa 2021-11-20T08:52:01.609 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T08:52:01.610 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.610 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T08:52:01.611 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Compile date May 13 2021 2021-11-20T08:52:01.611 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: DB SUMMARY 2021-11-20T08:52:01.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T08:52:01.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T08:52:01.613 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.613 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-11-20T08:52:01.613 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.613 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi093/store.db dir, Total Num: 1, files: 000004.sst 2021-11-20T08:52:01.613 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.614 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi093/store.db: 000006.log size: 92468 ; 2021-11-20T08:52:01.614 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.614 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T08:52:01.615 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T08:52:01.615 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T08:52:01.615 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.env: 0x55d91de2c1c0 2021-11-20T08:52:01.616 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.fs: Posix File System 2021-11-20T08:52:01.616 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.info_log: 0x55d920213960 2021-11-20T08:52:01.616 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T08:52:01.616 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.statistics: (nil) 2021-11-20T08:52:01.617 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.use_fsync: 0 2021-11-20T08:52:01.619 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T08:52:01.619 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T08:52:01.619 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T08:52:01.619 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T08:52:01.620 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T08:52:01.620 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.620 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T08:52:01.620 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T08:52:01.620 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T08:52:01.621 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T08:52:01.621 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T08:52:01.621 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T08:52:01.621 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.db_log_dir: 2021-11-20T08:52:01.622 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi093/store.db 2021-11-20T08:52:01.622 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T08:52:01.622 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T08:52:01.622 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T08:52:01.623 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T08:52:01.623 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.623 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T08:52:01.623 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T08:52:01.624 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T08:52:01.624 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T08:52:01.624 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T08:52:01.624 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T08:52:01.624 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.write_buffer_manager: 0x55d920e88030 2021-11-20T08:52:01.625 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T08:52:01.625 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T08:52:01.625 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T08:52:01.625 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T08:52:01.626 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T08:52:01.626 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.626 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T08:52:01.626 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T08:52:01.627 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T08:52:01.627 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T08:52:01.627 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.unordered_write: 0 2021-11-20T08:52:01.627 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T08:52:01.627 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T08:52:01.628 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T08:52:01.628 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T08:52:01.628 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.row_cache: None 2021-11-20T08:52:01.628 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.wal_filter: None 2021-11-20T08:52:01.629 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T08:52:01.629 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T08:52:01.629 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T08:52:01.629 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.630 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T08:52:01.630 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T08:52:01.630 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T08:52:01.630 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T08:52:01.631 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T08:52:01.631 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T08:52:01.631 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T08:52:01.631 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T08:52:01.631 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T08:52:01.632 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T08:52:01.632 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T08:52:01.632 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T08:52:01.632 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T08:52:01.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T08:52:01.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T08:52:01.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 bash[33405]: b1085d96938e7ee021ab2699cda5905feec6d3cdccca948308e9d01624c99c51 2021-11-20T08:52:01.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T08:52:01.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T08:52:01.634 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T08:52:01.634 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_open_files: -1 2021-11-20T08:52:01.634 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T08:52:01.634 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T08:52:01.635 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T08:52:01.635 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T08:52:01.635 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Compression algorithms supported: 2021-11-20T08:52:01.635 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T08:52:01.636 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kZSTD supported: 0 2021-11-20T08:52:01.636 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T08:52:01.636 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T08:52:01.636 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T08:52:01.636 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T08:52:01.637 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T08:52:01.637 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T08:52:01.637 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T08:52:01.637 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi093/store.db/MANIFEST-000005 2021-11-20T08:52:01.638 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.638 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T08:52:01.638 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.638 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T08:52:01.639 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.639 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.merge_operator: 2021-11-20T08:52:01.639 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compaction_filter: None 2021-11-20T08:52:01.639 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T08:52:01.640 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T08:52:01.640 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T08:52:01.640 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d92012a128) 2021-11-20T08:52:01.640 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: cache_index_and_filter_blocks: 1 2021-11-20T08:52:01.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T08:52:01.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T08:52:01.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: pin_top_level_index_and_filter: 1 2021-11-20T08:52:01.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: index_type: 0 2021-11-20T08:52:01.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: data_block_index_type: 0 2021-11-20T08:52:01.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: index_shortening: 1 2021-11-20T08:52:01.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T08:52:01.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: hash_index_allow_collision: 1 2021-11-20T08:52:01.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: checksum: 1 2021-11-20T08:52:01.643 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: no_block_cache: 0 2021-11-20T08:52:01.643 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_cache: 0x55d920160f10 2021-11-20T08:52:01.643 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_cache_name: BinnedLRUCache 2021-11-20T08:52:01.643 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_cache_options: 2021-11-20T08:52:01.644 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: capacity : 536870912 2021-11-20T08:52:01.644 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: num_shard_bits : 4 2021-11-20T08:52:01.644 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: strict_capacity_limit : 0 2021-11-20T08:52:01.644 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: high_pri_pool_ratio: 0.000 2021-11-20T08:52:01.644 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_cache_compressed: (nil) 2021-11-20T08:52:01.645 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: persistent_cache: (nil) 2021-11-20T08:52:01.645 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_size: 4096 2021-11-20T08:52:01.645 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_size_deviation: 10 2021-11-20T08:52:01.645 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_restart_interval: 16 2021-11-20T08:52:01.646 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: index_block_restart_interval: 1 2021-11-20T08:52:01.646 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: metadata_block_size: 4096 2021-11-20T08:52:01.646 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: partition_filters: 0 2021-11-20T08:52:01.646 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: use_delta_encoding: 1 2021-11-20T08:52:01.647 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T08:52:01.647 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: whole_key_filtering: 1 2021-11-20T08:52:01.647 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: verify_compression: 0 2021-11-20T08:52:01.647 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: read_amp_bytes_per_bit: 0 2021-11-20T08:52:01.647 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: format_version: 2 2021-11-20T08:52:01.648 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: enable_index_compression: 1 2021-11-20T08:52:01.648 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: block_align: 0 2021-11-20T08:52:01.648 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.648 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T08:52:01.649 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T08:52:01.649 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compression: NoCompression 2021-11-20T08:52:01.649 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T08:52:01.649 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T08:52:01.649 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T08:52:01.650 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.num_levels: 7 2021-11-20T08:52:01.650 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T08:52:01.650 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T08:52:01.650 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T08:52:01.651 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.651 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T08:52:01.651 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T08:52:01.651 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T08:52:01.652 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T08:52:01.652 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T08:52:01.652 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T08:52:01.652 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T08:52:01.653 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T08:52:01.653 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T08:52:01.653 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T08:52:01.653 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T08:52:01.653 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T08:52:01.654 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T08:52:01.654 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T08:52:01.654 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T08:52:01.654 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T08:52:01.655 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T08:52:01.655 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T08:52:01.655 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T08:52:01.655 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T08:52:01.655 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.444+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T08:52:01.656 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T08:52:01.656 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T08:52:01.656 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T08:52:01.656 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T08:52:01.657 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T08:52:01.657 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T08:52:01.657 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T08:52:01.657 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T08:52:01.658 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T08:52:01.658 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T08:52:01.658 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T08:52:01.658 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T08:52:01.658 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T08:52:01.659 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T08:52:01.659 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T08:52:01.659 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T08:52:01.659 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T08:52:01.660 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T08:52:01.660 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T08:52:01.660 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T08:52:01.660 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T08:52:01.661 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T08:52:01.661 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T08:52:01.661 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T08:52:01.661 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T08:52:01.661 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T08:52:01.662 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T08:52:01.662 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T08:52:01.662 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T08:52:01.662 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T08:52:01.663 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T08:52:01.663 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T08:52:01.663 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T08:52:01.663 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T08:52:01.663 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T08:52:01.664 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.ttl: 2592000 2021-11-20T08:52:01.664 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T08:52:01.664 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi093/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T08:52:01.664 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-11-20T08:52:01.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637398321446381, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-11-20T08:52:01.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.445+0000 7f018fd56700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-11-20T08:52:01.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.445+0000 7f018fd56700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T08:52:01.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.446+0000 7f018fd56700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637398321447219, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88268, "table_properties": {"data_size": 86395, "index_size": 350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12269, "raw_average_key_size": 49, "raw_value_size": 79609, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637398321, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T08:52:01.667 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.446+0000 7f018fd56700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-11-20T08:52:01.667 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.667 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.447+0000 7f018fd56700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637398321448099, "job": 1, "event": "recovery_finished"} 2021-11-20T08:52:01.667 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.447+0000 7f018fd56700 4 rocksdb: DB pointer 0x55d920225800 2021-11-20T08:52:01.668 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.448+0000 7f0176b6e700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T08:52:01.668 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.448+0000 7f0176b6e700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T08:52:01.668 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ** DB Stats ** 2021-11-20T08:52:01.668 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:52:01.668 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T08:52:01.669 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T08:52:01.669 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T08:52:01.669 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T08:52:01.669 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T08:52:01.670 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T08:52:01.670 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.670 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ** Compaction Stats [default] ** 2021-11-20T08:52:01.670 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:52:01.670 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:52:01.671 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: L0 2/0 87.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2021-11-20T08:52:01.671 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Sum 2/0 87.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2021-11-20T08:52:01.671 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2021-11-20T08:52:01.671 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.672 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ** Compaction Stats [default] ** 2021-11-20T08:52:01.672 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:52:01.672 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:52:01.672 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2021-11-20T08:52:01.673 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:52:01.673 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T08:52:01.673 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T08:52:01.673 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T08:52:01.674 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T08:52:01.674 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T08:52:01.674 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Cumulative compaction: 0.00 GB write, 27.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:52:01.674 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Interval compaction: 0.00 GB write, 27.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:52:01.674 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T08:52:01.675 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.675 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T08:52:01.675 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.675 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ** Compaction Stats [default] ** 2021-11-20T08:52:01.676 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:52:01.676 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:52:01.676 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: L0 2/0 87.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2021-11-20T08:52:01.676 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Sum 2/0 87.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2021-11-20T08:52:01.677 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T08:52:01.677 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.677 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ** Compaction Stats [default] ** 2021-11-20T08:52:01.677 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:52:01.677 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:52:01.678 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.3 0.00 0.00 1 0.001 0 0 2021-11-20T08:52:01.678 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:52:01.678 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T08:52:01.678 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T08:52:01.679 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T08:52:01.679 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T08:52:01.679 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T08:52:01.679 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Cumulative compaction: 0.00 GB write, 26.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:52:01.679 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:52:01.680 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T08:52:01.680 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.680 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T08:52:01.680 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.681 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.448+0000 7f018fd56700 0 starting mon.smithi093 rank 0 at public addrs [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] at bind addrs [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi093 fsid 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:52:01.681 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.449+0000 7f018fd56700 1 mon.smithi093@-1(???) e1 preinit fsid 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:52:01.681 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.449+0000 7f018fd56700 0 mon.smithi093@-1(???).mds e1 new map 2021-11-20T08:52:01.681 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.681 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.449+0000 7f018fd56700 0 mon.smithi093@-1(???).mds e1 print_map 2021-11-20T08:52:01.682 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: e1 2021-11-20T08:52:01.682 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T08:52:01.682 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T08:52:01.682 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: legacy client fscid: -1 2021-11-20T08:52:01.683 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.683 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: No filesystems configured 2021-11-20T08:52:01.683 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.683 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.449+0000 7f018fd56700 0 mon.smithi093@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T08:52:01.684 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.684 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:52:01.449+0000 7f018fd56700 0 mon.smithi093@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T08:52:01.684 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.449+0000 7f018fd56700 0 mon.smithi093@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T08:52:01.684 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.449+0000 7f018fd56700 0 mon.smithi093@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T08:52:01.685 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: audit 2021-11-20T08:52:01.685 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 2021-11-20T08:51:58. 2021-11-20T08:52:01.685 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 917036+0000 mon.smithi093 (mon.0) 8 2021-11-20T08:52:01.685 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: : audit [INF] from='client.? 172.21.15.93:0/2610969771' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-20T08:52:01.685 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:52:01.686 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 51:58.920248+0000 mon.smithi093 (mon. 2021-11-20T08:52:01.686 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: 0) 9 : audit [INF] from='client.? 172.21.15.93:0/2610969771' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-20T08:52:01.686 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.449+0000 7f018fd56700 1 mon.smithi093@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-11-20T08:52:01.686 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expand map: {default=false} 2021-11-20T08:52:01.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta from 'false' to 'false' 2021-11-20T08:52:01.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expanded map: {default=false} 2021-11-20T08:52:01.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expand map: {default=info} 2021-11-20T08:52:01.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta from 'info' to 'info' 2021-11-20T08:52:01.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expanded map: {default=info} 2021-11-20T08:52:01.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expand map: {default=daemon} 2021-11-20T08:52:01.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-11-20T08:52:01.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expanded map: {default=daemon} 2021-11-20T08:52:01.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expand map: {default=debug} 2021-11-20T08:52:01.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta from 'debug' to 'debug' 2021-11-20T08:52:01.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: debug 2021-11-20T08:52:01.450+0000 7f018fd56700 20 expand_channel_meta expanded map: {default=debug} 2021-11-20T08:52:01.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 systemd[1]: Started Ceph mon.smithi093 for 11458b34-49df-11ec-8c2c-001a4aab830c. 2021-11-20T08:52:01.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: cluster 2021-11-20T08:52:01.465369+0000 mon.smithi093 (mon 2021-11-20T08:52:01.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: .0) 1 : cluster [INF] mon.smithi093 is new leader, mons smithi093 in quorum (ranks 0) 2021-11-20T08:52:01.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: cluster 2021-11-20T08:52:01.465430+0000 mon.smithi093 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi093=[v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0]} 2021-11-20T08:52:01.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: cluster 2021-11-20T08:52:01.465461+0000 mon.smithi093 (mon.0) 3 : cluster [DBG] fsmap 2021-11-20T08:52:01.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: cluster 2021-11-20T08:52:01.465485+0000 mon.smithi093 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T08:52:01.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: cluster 2021-11-20T08:52:01.465845+0000 mon.smithi093 2021-11-20T08:52:01.691 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:01 smithi093 conmon[33522]: (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T08:52:01.930 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.926+0000 7f29b61be700 1 Processor -- start 2021-11-20T08:52:01.930 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.926+0000 7f29b61be700 1 -- start start 2021-11-20T08:52:01.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.926+0000 7f29b61be700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b0070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:01.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.926+0000 7f29b61be700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00720f0 con 0x7f29b0071cd0 2021-11-20T08:52:01.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.926+0000 7f29b51bc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:01.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.926+0000 7f29b51bc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b0070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42072/0 (socket says 172.21.15.93:42072) 2021-11-20T08:52:01.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.926+0000 7f29b51bc700 1 -- 172.21.15.93:0/3916119933 learned_addr learned my addr 172.21.15.93:0/3916119933 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:01.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.927+0000 7f29b51bc700 1 -- 172.21.15.93:0/3916119933 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0070760 con 0x7f29b0071cd0 2021-11-20T08:52:01.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.927+0000 7f29b51bc700 1 --2- 172.21.15.93:0/3916119933 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b0070270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f29ac00b5f0 tx=0x7f29ac006ee0).ready entity=mon.0 client_cookie=445fdad650b21097 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:01.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.927+0000 7f299ffff700 1 -- 172.21.15.93:0/3916119933 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ac00d600 con 0x7f29b0071cd0 2021-11-20T08:52:01.933 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.927+0000 7f299ffff700 1 -- 172.21.15.93:0/3916119933 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29ac00dbd0 con 0x7f29b0071cd0 2021-11-20T08:52:01.933 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.927+0000 7f299ffff700 1 -- 172.21.15.93:0/3916119933 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ac011c00 con 0x7f29b0071cd0 2021-11-20T08:52:01.934 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 -- 172.21.15.93:0/3916119933 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 msgr2=0x7f29b0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:01.935 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 --2- 172.21.15.93:0/3916119933 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b0070270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f29ac00b5f0 tx=0x7f29ac006ee0).stop 2021-11-20T08:52:01.937 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 -- 172.21.15.93:0/3916119933 shutdown_connections 2021-11-20T08:52:01.937 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 --2- 172.21.15.93:0/3916119933 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b0070270 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:01.938 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 -- 172.21.15.93:0/3916119933 >> 172.21.15.93:0/3916119933 conn(0x7f29b00f71e0 msgr2=0x7f29b00f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:01.938 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 -- 172.21.15.93:0/3916119933 shutdown_connections 2021-11-20T08:52:01.938 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 -- 172.21.15.93:0/3916119933 wait complete. 2021-11-20T08:52:01.938 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 Processor -- start 2021-11-20T08:52:01.939 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.928+0000 7f29b61be700 1 -- start start 2021-11-20T08:52:01.939 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b61be700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b010c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:01.939 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b61be700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b010cee0 con 0x7f29b0071cd0 2021-11-20T08:52:01.939 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b51bc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b010c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b51bc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b010c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42074/0 (socket says 172.21.15.93:42074) 2021-11-20T08:52:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b51bc700 1 -- 172.21.15.93:0/3043410382 learned_addr learned my addr 172.21.15.93:0/3043410382 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b51bc700 1 -- 172.21.15.93:0/3043410382 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac00b200 con 0x7f29b0071cd0 2021-11-20T08:52:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b51bc700 1 --2- 172.21.15.93:0/3043410382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b010c9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f29ac00e040 tx=0x7f29ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f299dffb700 1 -- 172.21.15.93:0/3043410382 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ac00d9f0 con 0x7f29b0071cd0 2021-11-20T08:52:01.941 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b010d0e0 con 0x7f29b0071cd0 2021-11-20T08:52:01.941 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f299dffb700 1 -- 172.21.15.93:0/3043410382 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29ac01a9f0 con 0x7f29b0071cd0 2021-11-20T08:52:01.941 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f299dffb700 1 -- 172.21.15.93:0/3043410382 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ac01b8c0 con 0x7f29b0071cd0 2021-11-20T08:52:01.941 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.929+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b010d4f0 con 0x7f29b0071cd0 2021-11-20T08:52:01.941 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.930+0000 7f299dffb700 1 -- 172.21.15.93:0/3043410382 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f29ac00f070 con 0x7f29b0071cd0 2021-11-20T08:52:01.942 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.930+0000 7f299dffb700 1 -- 172.21.15.93:0/3043410382 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f29ac025b30 con 0x7f29b0071cd0 2021-11-20T08:52:01.942 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.930+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b0045ad0 con 0x7f29b0071cd0 2021-11-20T08:52:01.943 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.933+0000 7f299dffb700 1 -- 172.21.15.93:0/3043410382 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f29ac016070 con 0x7f29b0071cd0 2021-11-20T08:52:01.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.995+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f29b0106810 con 0x7f29b0071cd0 2021-11-20T08:52:01.997 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.998+0000 7f299dffb700 1 -- 172.21.15.93:0/3043410382 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f29ac01bdc0 con 0x7f29b0071cd0 2021-11-20T08:52:01.998 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.999+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 msgr2=0x7f29b010c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:01.998 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.999+0000 7f29b61be700 1 --2- 172.21.15.93:0/3043410382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b010c9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f29ac00e040 tx=0x7f29ac006ee0).stop 2021-11-20T08:52:01.999 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.999+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 shutdown_connections 2021-11-20T08:52:01.999 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.999+0000 7f29b61be700 1 --2- 172.21.15.93:0/3043410382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29b0071cd0 0x7f29b010c9f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:01.999 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.999+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 >> 172.21.15.93:0/3043410382 conn(0x7f29b00f71e0 msgr2=0x7f29b00f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:01.999 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.999+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 shutdown_connections 2021-11-20T08:52:02.000 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:01.999+0000 7f29b61be700 1 -- 172.21.15.93:0/3043410382 wait complete. 2021-11-20T08:52:02.310 INFO:teuthology.orchestra.run.smithi093.stderr:Wrote config to /etc/ceph/ceph.conf 2021-11-20T08:52:02.311 INFO:teuthology.orchestra.run.smithi093.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-11-20T08:52:02.312 INFO:teuthology.orchestra.run.smithi093.stderr:Creating mgr... 2021-11-20T08:52:02.312 INFO:teuthology.orchestra.run.smithi093.stderr:Verifying port 9283 ... 2021-11-20T08:52:02.313 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl daemon-reload 2021-11-20T08:52:02.402 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mgr.smithi093.qdikkm 2021-11-20T08:52:02.420 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl reset-failed ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mgr.smithi093.qdikkm 2021-11-20T08:52:02.425 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Failed to reset failed state of unit ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mgr.smithi093.qdikkm.service: Unit ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mgr.smithi093.qdikkm.service not loaded. 2021-11-20T08:52:02.426 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl enable ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mgr.smithi093.qdikkm 2021-11-20T08:52:02.431 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: Created symlink /etc/systemd/system/ceph-11458b34-49df-11ec-8c2c-001a4aab830c.target.wants/ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mgr.smithi093.qdikkm.service → /etc/systemd/system/ceph-11458b34-49df-11ec-8c2c-001a4aab830c@.service. 2021-11-20T08:52:02.508 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl start ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mgr.smithi093.qdikkm 2021-11-20T08:52:03.038 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T08:52:03.046 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: disabled 2021-11-20T08:52:03.046 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T08:52:03.053 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: inactive 2021-11-20T08:52:03.054 INFO:teuthology.orchestra.run.smithi093.stderr:firewalld.service is not enabled 2021-11-20T08:52:03.054 INFO:teuthology.orchestra.run.smithi093.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T08:52:03.054 INFO:teuthology.orchestra.run.smithi093.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T08:52:03.055 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T08:52:03.061 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: disabled 2021-11-20T08:52:03.062 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T08:52:03.068 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: inactive 2021-11-20T08:52:03.069 INFO:teuthology.orchestra.run.smithi093.stderr:firewalld.service is not enabled 2021-11-20T08:52:03.069 INFO:teuthology.orchestra.run.smithi093.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-11-20T08:52:03.070 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for mgr to start... 2021-11-20T08:52:03.070 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for mgr... 2021-11-20T08:52:03.070 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T08:52:03.261 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:02 smithi093 conmon[33522]: audit 2021-11-20T08:52:01.999195+0000 mon.smithi093 (mon.0) 6 : audit [INF] from='client.? 172.21.15.93:0/3043410382' entity='client.admin' 2021-11-20T08:52:03.525 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.521+0000 7fec18c45700 1 Processor -- start 2021-11-20T08:52:03.526 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.522+0000 7fec18c45700 1 -- start start 2021-11-20T08:52:03.526 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.522+0000 7fec18c45700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec14100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:03.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.522+0000 7fec18c45700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140fc6a0 con 0x7fec1406fbb0 2021-11-20T08:52:03.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.522+0000 7fec1259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec14100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:03.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.522+0000 7fec1259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec14100cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42080/0 (socket says 172.21.15.93:42080) 2021-11-20T08:52:03.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.522+0000 7fec1259c700 1 -- 172.21.15.93:0/2500912394 learned_addr learned my addr 172.21.15.93:0/2500912394 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:03.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.523+0000 7fec1259c700 1 -- 172.21.15.93:0/2500912394 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec141011a0 con 0x7fec1406fbb0 2021-11-20T08:52:03.528 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.523+0000 7fec1259c700 1 --2- 172.21.15.93:0/2500912394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec14100cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fec0800b5f0 tx=0x7fec08006ee0).ready entity=mon.0 client_cookie=29aadb8479b8efd7 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:03.528 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.523+0000 7fec1159a700 1 -- 172.21.15.93:0/2500912394 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0800d600 con 0x7fec1406fbb0 2021-11-20T08:52:03.528 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.523+0000 7fec1159a700 1 -- 172.21.15.93:0/2500912394 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fec0800dbd0 con 0x7fec1406fbb0 2021-11-20T08:52:03.528 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.523+0000 7fec1159a700 1 -- 172.21.15.93:0/2500912394 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec08011c00 con 0x7fec1406fbb0 2021-11-20T08:52:03.529 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.524+0000 7fec18c45700 1 -- 172.21.15.93:0/2500912394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 msgr2=0x7fec14100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:03.529 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.524+0000 7fec18c45700 1 --2- 172.21.15.93:0/2500912394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec14100cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fec0800b5f0 tx=0x7fec08006ee0).stop 2021-11-20T08:52:03.529 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.524+0000 7fec18c45700 1 -- 172.21.15.93:0/2500912394 shutdown_connections 2021-11-20T08:52:03.529 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.524+0000 7fec18c45700 1 --2- 172.21.15.93:0/2500912394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec14100cb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:03.530 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.524+0000 7fec18c45700 1 -- 172.21.15.93:0/2500912394 >> 172.21.15.93:0/2500912394 conn(0x7fec140f6310 msgr2=0x7fec140f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:03.530 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.524+0000 7fec18c45700 1 -- 172.21.15.93:0/2500912394 shutdown_connections 2021-11-20T08:52:03.530 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.524+0000 7fec18c45700 1 -- 172.21.15.93:0/2500912394 wait complete. 2021-11-20T08:52:03.530 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec18c45700 1 Processor -- start 2021-11-20T08:52:03.531 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec18c45700 1 -- start start 2021-11-20T08:52:03.531 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec18c45700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec1410cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:03.531 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec18c45700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1410d0b0 con 0x7fec1406fbb0 2021-11-20T08:52:03.531 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec1259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec1410cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:03.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec1259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec1410cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42082/0 (socket says 172.21.15.93:42082) 2021-11-20T08:52:03.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec1259c700 1 -- 172.21.15.93:0/488553812 learned_addr learned my addr 172.21.15.93:0/488553812 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:03.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec1259c700 1 -- 172.21.15.93:0/488553812 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0800b200 con 0x7fec1406fbb0 2021-11-20T08:52:03.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec1259c700 1 --2- 172.21.15.93:0/488553812 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec1410cbc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fec0800e040 tx=0x7fec08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:03.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.525+0000 7fec02ffd700 1 -- 172.21.15.93:0/488553812 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0800d9f0 con 0x7fec1406fbb0 2021-11-20T08:52:03.533 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.526+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1410d2b0 con 0x7fec1406fbb0 2021-11-20T08:52:03.533 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.526+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1410d6c0 con 0x7fec1406fbb0 2021-11-20T08:52:03.533 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.526+0000 7fec02ffd700 1 -- 172.21.15.93:0/488553812 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fec0801a9f0 con 0x7fec1406fbb0 2021-11-20T08:52:03.533 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.526+0000 7fec02ffd700 1 -- 172.21.15.93:0/488553812 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0801b8c0 con 0x7fec1406fbb0 2021-11-20T08:52:03.534 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.526+0000 7fec02ffd700 1 -- 172.21.15.93:0/488553812 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fec0800f070 con 0x7fec1406fbb0 2021-11-20T08:52:03.534 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.526+0000 7fec02ffd700 1 -- 172.21.15.93:0/488553812 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fec08025b90 con 0x7fec1406fbb0 2021-11-20T08:52:03.535 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.527+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf0004fa0 con 0x7fec1406fbb0 2021-11-20T08:52:03.535 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.529+0000 7fec02ffd700 1 -- 172.21.15.93:0/488553812 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fec08016070 con 0x7fec1406fbb0 2021-11-20T08:52:03.606 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.597+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7febf0005940 con 0x7fec1406fbb0 2021-11-20T08:52:03.607 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.597+0000 7fec02ffd700 1 -- 172.21.15.93:0/488553812 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fec0801ab60 con 0x7fec1406fbb0 2021-11-20T08:52:03.607 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.608 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: { 2021-11-20T08:52:03.608 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "fsid": "11458b34-49df-11ec-8c2c-001a4aab830c", 2021-11-20T08:52:03.609 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "health": { 2021-11-20T08:52:03.609 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T08:52:03.609 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T08:52:03.610 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T08:52:03.610 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:03.610 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T08:52:03.611 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T08:52:03.611 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 0 2021-11-20T08:52:03.612 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:03.612 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T08:52:03.613 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "smithi093" 2021-11-20T08:52:03.614 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:03.615 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-11-20T08:52:03.615 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T08:52:03.615 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:03.615 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T08:52:03.616 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T08:52:03.616 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:03.617 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T08:52:03.617 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:03.618 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T08:52:03.619 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T08:52:03.619 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T08:52:03.619 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T08:52:03.619 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T08:52:03.620 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T08:52:03.620 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:03.620 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T08:52:03.620 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T08:52:03.620 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T08:52:03.621 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T08:52:03.621 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T08:52:03.621 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T08:52:03.621 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T08:52:03.622 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T08:52:03.622 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T08:52:03.622 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:03.622 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T08:52:03.623 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:03.623 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T08:52:03.623 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T08:52:03.623 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:03.624 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T08:52:03.624 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "available": false, 2021-11-20T08:52:03.624 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T08:52:03.624 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T08:52:03.625 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "iostat", 2021-11-20T08:52:03.626 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "restful" 2021-11-20T08:52:03.626 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:03.627 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "services": {} 2021-11-20T08:52:03.627 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:03.627 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T08:52:03.627 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:03.627 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "modified": "2021-11-20T08:51:57.464883+0000", 2021-11-20T08:52:03.628 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "services": {} 2021-11-20T08:52:03.628 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:03.628 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T08:52:03.630 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: } 2021-11-20T08:52:03.630 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.599+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 msgr2=0x7fec1410cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:03.631 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.599+0000 7fec18c45700 1 --2- 172.21.15.93:0/488553812 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec1410cbc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fec0800e040 tx=0x7fec08006ee0).stop 2021-11-20T08:52:03.631 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.599+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 shutdown_connections 2021-11-20T08:52:03.631 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.599+0000 7fec18c45700 1 --2- 172.21.15.93:0/488553812 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec1406fbb0 0x7fec1410cbc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:03.632 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.599+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 >> 172.21.15.93:0/488553812 conn(0x7fec140f6310 msgr2=0x7fec140f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:03.632 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.599+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 shutdown_connections 2021-11-20T08:52:03.633 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:03.599+0000 7fec18c45700 1 -- 172.21.15.93:0/488553812 wait complete. 2021-11-20T08:52:03.910 INFO:teuthology.orchestra.run.smithi093.stderr:mgr not available, waiting (1/15)... 2021-11-20T08:52:04.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:03 smithi093 conmon[33522]: audit 2021-11-20T08:52:03.598508+0000 mon.smithi093 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.93:0/488553812' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T08:52:05.912 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T08:52:06.358 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.357+0000 7f99ae268700 1 Processor -- start 2021-11-20T08:52:06.358 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.357+0000 7f99ae268700 1 -- start start 2021-11-20T08:52:06.358 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.357+0000 7f99ae268700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:06.359 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.357+0000 7f99ae268700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80fed60 con 0x7f99a806fc90 2021-11-20T08:52:06.359 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.357+0000 7f99a7fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:06.359 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.357+0000 7f99a7fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42084/0 (socket says 172.21.15.93:42084) 2021-11-20T08:52:06.359 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.357+0000 7f99a7fff700 1 -- 172.21.15.93:0/2368835620 learned_addr learned my addr 172.21.15.93:0/2368835620 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:06.359 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.358+0000 7f99a7fff700 1 -- 172.21.15.93:0/2368835620 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a80feea0 con 0x7f99a806fc90 2021-11-20T08:52:06.360 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.358+0000 7f99a7fff700 1 --2- 172.21.15.93:0/2368835620 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fe7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f999800b5f0 tx=0x7f9998006ee0).ready entity=mon.0 client_cookie=9ec4da4220998d20 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:06.360 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.358+0000 7f99a6ffd700 1 -- 172.21.15.93:0/2368835620 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f999800d600 con 0x7f99a806fc90 2021-11-20T08:52:06.360 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.358+0000 7f99a6ffd700 1 -- 172.21.15.93:0/2368835620 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f999800dbd0 con 0x7f99a806fc90 2021-11-20T08:52:06.360 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.358+0000 7f99a6ffd700 1 -- 172.21.15.93:0/2368835620 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9998011c00 con 0x7f99a806fc90 2021-11-20T08:52:06.361 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.358+0000 7f99ae268700 1 -- 172.21.15.93:0/2368835620 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 msgr2=0x7f99a80fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:06.361 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.358+0000 7f99ae268700 1 --2- 172.21.15.93:0/2368835620 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fe7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f999800b5f0 tx=0x7f9998006ee0).stop 2021-11-20T08:52:06.361 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.359+0000 7f99ae268700 1 -- 172.21.15.93:0/2368835620 shutdown_connections 2021-11-20T08:52:06.361 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.359+0000 7f99ae268700 1 --2- 172.21.15.93:0/2368835620 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fe7e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:06.363 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.359+0000 7f99ae268700 1 -- 172.21.15.93:0/2368835620 >> 172.21.15.93:0/2368835620 conn(0x7f99a80f6280 msgr2=0x7f99a80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:06.363 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.359+0000 7f99ae268700 1 -- 172.21.15.93:0/2368835620 shutdown_connections 2021-11-20T08:52:06.363 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.359+0000 7f99ae268700 1 -- 172.21.15.93:0/2368835620 wait complete. 2021-11-20T08:52:06.364 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.359+0000 7f99ae268700 1 Processor -- start 2021-11-20T08:52:06.364 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.359+0000 7f99ae268700 1 -- start start 2021-11-20T08:52:06.364 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99ae268700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:06.364 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99ae268700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80fa6e0 con 0x7f99a806fc90 2021-11-20T08:52:06.364 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a7fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:06.365 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a7fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42086/0 (socket says 172.21.15.93:42086) 2021-11-20T08:52:06.365 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a7fff700 1 -- 172.21.15.93:0/2060466908 learned_addr learned my addr 172.21.15.93:0/2060466908 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:06.365 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a7fff700 1 -- 172.21.15.93:0/2060466908 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999800b200 con 0x7f99a806fc90 2021-11-20T08:52:06.365 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a7fff700 1 --2- 172.21.15.93:0/2060466908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fc260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f999800e040 tx=0x7f9998006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:06.366 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a4ff9700 1 -- 172.21.15.93:0/2060466908 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f999801b410 con 0x7f99a806fc90 2021-11-20T08:52:06.366 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a80fa880 con 0x7f99a806fc90 2021-11-20T08:52:06.366 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a4ff9700 1 -- 172.21.15.93:0/2060466908 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f999801b9e0 con 0x7f99a806fc90 2021-11-20T08:52:06.366 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99a4ff9700 1 -- 172.21.15.93:0/2060466908 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f999801a770 con 0x7f99a806fc90 2021-11-20T08:52:06.366 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.360+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a80fac90 con 0x7f99a806fc90 2021-11-20T08:52:06.367 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.361+0000 7f99a4ff9700 1 -- 172.21.15.93:0/2060466908 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f999800f070 con 0x7f99a806fc90 2021-11-20T08:52:06.367 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.361+0000 7f99a4ff9700 1 -- 172.21.15.93:0/2060466908 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9998024a80 con 0x7f99a806fc90 2021-11-20T08:52:06.367 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.361+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a8045ad0 con 0x7f99a806fc90 2021-11-20T08:52:06.367 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.363+0000 7f99a4ff9700 1 -- 172.21.15.93:0/2060466908 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9998016070 con 0x7f99a806fc90 2021-11-20T08:52:06.431 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.431+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f99a80008d0 con 0x7f99a806fc90 2021-11-20T08:52:06.431 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.432+0000 7f99a4ff9700 1 -- 172.21.15.93:0/2060466908 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9998024cc0 con 0x7f99a806fc90 2021-11-20T08:52:06.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: { 2021-11-20T08:52:06.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "fsid": "11458b34-49df-11ec-8c2c-001a4aab830c", 2021-11-20T08:52:06.438 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "health": { 2021-11-20T08:52:06.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T08:52:06.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T08:52:06.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T08:52:06.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:06.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T08:52:06.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T08:52:06.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 0 2021-11-20T08:52:06.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:06.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T08:52:06.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "smithi093" 2021-11-20T08:52:06.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:06.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum_age": 4, 2021-11-20T08:52:06.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T08:52:06.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:06.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T08:52:06.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T08:52:06.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:06.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T08:52:06.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:06.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T08:52:06.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T08:52:06.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T08:52:06.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T08:52:06.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T08:52:06.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T08:52:06.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:06.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T08:52:06.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T08:52:06.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T08:52:06.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T08:52:06.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T08:52:06.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T08:52:06.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T08:52:06.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T08:52:06.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T08:52:06.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:06.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T08:52:06.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:06.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T08:52:06.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T08:52:06.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:06.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T08:52:06.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "available": false, 2021-11-20T08:52:06.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T08:52:06.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T08:52:06.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "iostat", 2021-11-20T08:52:06.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "restful" 2021-11-20T08:52:06.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:06.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "services": {} 2021-11-20T08:52:06.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:06.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T08:52:06.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:06.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "modified": "2021-11-20T08:51:57.464883+0000", 2021-11-20T08:52:06.452 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "services": {} 2021-11-20T08:52:06.452 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:06.452 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T08:52:06.452 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: } 2021-11-20T08:52:06.452 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.433+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 msgr2=0x7f99a80fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:06.453 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.433+0000 7f99ae268700 1 --2- 172.21.15.93:0/2060466908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fc260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f999800e040 tx=0x7f9998006ee0).stop 2021-11-20T08:52:06.453 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.433+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 shutdown_connections 2021-11-20T08:52:06.453 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.433+0000 7f99ae268700 1 --2- 172.21.15.93:0/2060466908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f99a806fc90 0x7f99a80fc260 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:06.453 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.433+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 >> 172.21.15.93:0/2060466908 conn(0x7f99a80f6280 msgr2=0x7f99a80f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:06.454 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.433+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 shutdown_connections 2021-11-20T08:52:06.454 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:06.433+0000 7f99ae268700 1 -- 172.21.15.93:0/2060466908 wait complete. 2021-11-20T08:52:06.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:06 smithi093 conmon[33522]: audit 2021-11-20T08:52:06.433051+0000 mon.smithi093 (mon. 2021-11-20T08:52:06.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:06 smithi093 conmon[33522]: 0) 8 : audit [DBG] from='client.? 172.21.15.93:0/2060466908' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T08:52:06.949 INFO:teuthology.orchestra.run.smithi093.stderr:mgr not available, waiting (2/15)... 2021-11-20T08:52:08.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: cluster 2021-11-20T08:52:07.895670+0000 mon.smithi093 (mon 2021-11-20T08:52:08.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: .0) 9 : cluster [INF] Activating manager daemon smithi093.qdikkm 2021-11-20T08:52:08.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: cluster 2021-11-20T08:52:07.898129+0000 mon.smithi093 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi093.qdikkm(active, starting, since 0.00256301s) 2021-11-20T08:52:08.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11-20T08:52:07.899012+0000 mon.smithi093 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T08:52:08.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11-20T08:52:07.899173+0000 mon.smithi093 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T08:52:08.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11-20T08:52:07.899373+0000 mon.smithi093 (mon.0) 13 : audit [DBG] 2021-11-20T08:52:08.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T08:52:08.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11-20T08:52:07 2021-11-20T08:52:08.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: .899563+0000 mon.smithi093 (mon.0) 14 : audit 2021-11-20T08:52:08.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: [DBG] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi093"}]: dispatch 2021-11-20T08:52:08.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11-20T08:52 2021-11-20T08:52:08.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: :07.899750+0000 mon.smithi093 (mon.0) 15 2021-11-20T08:52:08.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: : audit [DBG] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr metadata", "who": "smithi093.qdikkm", "id": "smithi093.qdikkm"}]: dispatch 2021-11-20T08:52:08.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:52:08.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 08:52:07.913282+0000 mon.smithi093 (mon. 2021-11-20T08:52:08.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 0) 16 : cluster [INF] Manager daemon smithi093.qdikkm is now available 2021-11-20T08:52:08.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021 2021-11-20T08:52:08.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: -11-20T08:52:07.922454+0000 2021-11-20T08:52:08.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: mon.smithi093 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:08.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 2021-11-20T08:52:08.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11-20T08:52:07. 2021-11-20T08:52:08.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 925605+0000 mon.smithi093 (mon.0) 18 : audit 2021-11-20T08:52:08.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: [INF] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:52:08.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:52:08.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 52:07.926699+0000 mon.smithi093 (mon.0) 2021-11-20T08:52:08.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 19 : audit [INF] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:52:08.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:52:08.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 20T08:52:07.931017+0000 mon.smithi093 ( 2021-11-20T08:52:08.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:08.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:52:08.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 11-20T08:52:07.932722+0000 mon.smithi093 2021-11-20T08:52:08.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.93:0/3998551762' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:08.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:07 smithi093 conmon[33522]: 2021-11-20T08:52:08.952 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T08:52:09.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.437+0000 7f7e72ad0700 1 Processor -- start 2021-11-20T08:52:09.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.437+0000 7f7e72ad0700 1 -- start start 2021-11-20T08:52:09.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.437+0000 7f7e72ad0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c0fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:09.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.437+0000 7f7e72ad0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c0fed20 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e7086c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c0fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:09.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e7086c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c0fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42106/0 (socket says 172.21.15.93:42106) 2021-11-20T08:52:09.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e7086c700 1 -- 172.21.15.93:0/3359640066 learned_addr learned my addr 172.21.15.93:0/3359640066 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:09.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e7086c700 1 -- 172.21.15.93:0/3359640066 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0fee60 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e7086c700 1 --2- 172.21.15.93:0/3359640066 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c0fe7a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7e6800b5f0 tx=0x7f7e68006ee0).ready entity=mon.0 client_cookie=b97bddbd8048dcba server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:09.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e637fe700 1 -- 172.21.15.93:0/3359640066 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e6800d600 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e637fe700 1 -- 172.21.15.93:0/3359640066 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7e6800dbd0 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.438+0000 7f7e637fe700 1 -- 172.21.15.93:0/3359640066 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e68011c00 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 -- 172.21.15.93:0/3359640066 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 msgr2=0x7f7e6c0fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:09.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 --2- 172.21.15.93:0/3359640066 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c0fe7a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7e6800b5f0 tx=0x7f7e68006ee0).stop 2021-11-20T08:52:09.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 -- 172.21.15.93:0/3359640066 shutdown_connections 2021-11-20T08:52:09.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 --2- 172.21.15.93:0/3359640066 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c0fe7a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:09.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 -- 172.21.15.93:0/3359640066 >> 172.21.15.93:0/3359640066 conn(0x7f7e6c0f6280 msgr2=0x7f7e6c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:09.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 -- 172.21.15.93:0/3359640066 shutdown_connections 2021-11-20T08:52:09.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 -- 172.21.15.93:0/3359640066 wait complete. 2021-11-20T08:52:09.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 Processor -- start 2021-11-20T08:52:09.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.439+0000 7f7e72ad0700 1 -- start start 2021-11-20T08:52:09.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e72ad0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c10a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:09.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e72ad0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c10aee0 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e7086c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c10a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:09.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e7086c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c10a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42108/0 (socket says 172.21.15.93:42108) 2021-11-20T08:52:09.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e7086c700 1 -- 172.21.15.93:0/419173484 learned_addr learned my addr 172.21.15.93:0/419173484 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:09.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e7086c700 1 -- 172.21.15.93:0/419173484 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6800b200 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e7086c700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c10a9f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7e68000c00 tx=0x7f7e68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:09.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e617fa700 1 -- 172.21.15.93:0/419173484 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e68011690 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.440+0000 7f7e617fa700 1 -- 172.21.15.93:0/419173484 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7e6800d9f0 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.441+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c10b0e0 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.441+0000 7f7e617fa700 1 -- 172.21.15.93:0/419173484 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e6801b710 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.441+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6c10b4f0 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.441+0000 7f7e617fa700 1 -- 172.21.15.93:0/419173484 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f7e6800f070 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.441+0000 7f7e617fa700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f7e5802fca0 0x7f7e58032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:09.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.441+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6c045ad0 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.442+0000 7f7e617fa700 1 -- 172.21.15.93:0/419173484 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7e68042930 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.442+0000 7f7e63fff700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f7e5802fca0 0x7f7e58032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:09.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.442+0000 7f7e63fff700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f7e5802fca0 0x7f7e58032160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7e5c0096a0 tx=0x7f7e5c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:09.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.446+0000 7f7e617fa700 1 -- 172.21.15.93:0/419173484 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e68047060 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.662 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.648+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7e6c10c5a0 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.663 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.649+0000 7f7e617fa700 1 -- 172.21.15.93:0/419173484 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f7e68016070 con 0x7f7e6c06fcd0 2021-11-20T08:52:09.663 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.663 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: { 2021-11-20T08:52:09.664 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "fsid": "11458b34-49df-11ec-8c2c-001a4aab830c", 2021-11-20T08:52:09.664 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "health": { 2021-11-20T08:52:09.664 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T08:52:09.664 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T08:52:09.664 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T08:52:09.665 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:09.665 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T08:52:09.665 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T08:52:09.665 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 0 2021-11-20T08:52:09.666 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:09.666 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T08:52:09.666 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "smithi093" 2021-11-20T08:52:09.666 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:09.666 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "quorum_age": 8, 2021-11-20T08:52:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T08:52:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T08:52:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T08:52:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T08:52:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T08:52:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T08:52:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T08:52:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T08:52:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T08:52:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T08:52:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T08:52:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T08:52:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T08:52:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T08:52:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T08:52:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T08:52:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T08:52:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T08:52:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T08:52:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T08:52:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T08:52:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T08:52:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:09.674 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T08:52:09.674 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "available": true, 2021-11-20T08:52:09.674 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T08:52:09.674 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T08:52:09.674 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "iostat", 2021-11-20T08:52:09.675 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "restful" 2021-11-20T08:52:09.675 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ], 2021-11-20T08:52:09.675 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "services": {} 2021-11-20T08:52:09.675 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:09.676 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T08:52:09.676 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T08:52:09.676 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "modified": "2021-11-20T08:51:57.464883+0000", 2021-11-20T08:52:09.676 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "services": {} 2021-11-20T08:52:09.676 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: }, 2021-11-20T08:52:09.677 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T08:52:09.677 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: } 2021-11-20T08:52:09.677 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f7e5802fca0 msgr2=0x7f7e58032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:09.677 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f7e5802fca0 0x7f7e58032160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7e5c0096a0 tx=0x7f7e5c006b40).stop 2021-11-20T08:52:09.678 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 msgr2=0x7f7e6c10a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:09.678 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c10a9f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7e68000c00 tx=0x7f7e68006ee0).stop 2021-11-20T08:52:09.678 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 shutdown_connections 2021-11-20T08:52:09.678 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f7e5802fca0 0x7f7e58032160 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:09.678 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 --2- 172.21.15.93:0/419173484 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e6c06fcd0 0x7f7e6c10a9f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:09.679 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 >> 172.21.15.93:0/419173484 conn(0x7f7e6c0f6280 msgr2=0x7f7e6c0f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:09.679 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 shutdown_connections 2021-11-20T08:52:09.679 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:09.651+0000 7f7e72ad0700 1 -- 172.21.15.93:0/419173484 wait complete. 2021-11-20T08:52:09.993 INFO:teuthology.orchestra.run.smithi093.stderr:mgr is available 2021-11-20T08:52:09.993 INFO:teuthology.orchestra.run.smithi093.stderr:Enabling cephadm module... 2021-11-20T08:52:09.993 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2021-11-20T08:52:10.141 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:09 smithi093 conmon[33522]: cluster 2021-11-20T08:52:08.900640+0000 mon.smithi093 (mon.0) 22 : cluster [DBG] 2021-11-20T08:52:10.141 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:09 smithi093 conmon[33522]: mgrmap e3: smithi093.qdikkm(active, since 1.00507s) 2021-11-20T08:52:10.141 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:09 smithi093 conmon[33522]: audit 2021-11-20T08:52:09.649744+0000 mon.smithi093 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.93:0/419173484' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T08:52:10.429 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.425+0000 7f88ac74c700 1 Processor -- start 2021-11-20T08:52:10.429 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.425+0000 7f88ac74c700 1 -- start start 2021-11-20T08:52:10.429 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.425+0000 7f88ac74c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a4070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:10.430 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.425+0000 7f88ac74c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a40720b0 con 0x7f88a4071c90 2021-11-20T08:52:10.430 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.425+0000 7f88aa4e8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a4070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:10.430 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.425+0000 7f88aa4e8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a4070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42112/0 (socket says 172.21.15.93:42112) 2021-11-20T08:52:10.431 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.425+0000 7f88aa4e8700 1 -- 172.21.15.93:0/3571692075 learned_addr learned my addr 172.21.15.93:0/3571692075 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:10.431 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88aa4e8700 1 -- 172.21.15.93:0/3571692075 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4070720 con 0x7f88a4071c90 2021-11-20T08:52:10.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88aa4e8700 1 --2- 172.21.15.93:0/3571692075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a4070230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f88a000b5f0 tx=0x7f88a0006ee0).ready entity=mon.0 client_cookie=9082d1f63743f2e2 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:10.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88a94e6700 1 -- 172.21.15.93:0/3571692075 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a000d600 con 0x7f88a4071c90 2021-11-20T08:52:10.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88a94e6700 1 -- 172.21.15.93:0/3571692075 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f88a000dbd0 con 0x7f88a4071c90 2021-11-20T08:52:10.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88a94e6700 1 -- 172.21.15.93:0/3571692075 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a0011c00 con 0x7f88a4071c90 2021-11-20T08:52:10.433 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88ac74c700 1 -- 172.21.15.93:0/3571692075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 msgr2=0x7f88a4070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:10.434 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88ac74c700 1 --2- 172.21.15.93:0/3571692075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a4070230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f88a000b5f0 tx=0x7f88a0006ee0).stop 2021-11-20T08:52:10.434 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88ac74c700 1 -- 172.21.15.93:0/3571692075 shutdown_connections 2021-11-20T08:52:10.434 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88ac74c700 1 --2- 172.21.15.93:0/3571692075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a4070230 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:10.434 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88ac74c700 1 -- 172.21.15.93:0/3571692075 >> 172.21.15.93:0/3571692075 conn(0x7f88a40f73e0 msgr2=0x7f88a40f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:10.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.426+0000 7f88ac74c700 1 -- 172.21.15.93:0/3571692075 shutdown_connections 2021-11-20T08:52:10.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88ac74c700 1 -- 172.21.15.93:0/3571692075 wait complete. 2021-11-20T08:52:10.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88ac74c700 1 Processor -- start 2021-11-20T08:52:10.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88ac74c700 1 -- start start 2021-11-20T08:52:10.436 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88ac74c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a410ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:10.436 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88ac74c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a410d1b0 con 0x7f88a4071c90 2021-11-20T08:52:10.436 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88aa4e8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a410ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:10.436 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88aa4e8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a410ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42114/0 (socket says 172.21.15.93:42114) 2021-11-20T08:52:10.436 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.427+0000 7f88aa4e8700 1 -- 172.21.15.93:0/1814636184 learned_addr learned my addr 172.21.15.93:0/1814636184 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:10.437 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.428+0000 7f88aa4e8700 1 -- 172.21.15.93:0/1814636184 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a000b200 con 0x7f88a4071c90 2021-11-20T08:52:10.437 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.428+0000 7f88aa4e8700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a410ccc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f88a0006c70 tx=0x7f88a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:10.437 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.428+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a000da30 con 0x7f88a4071c90 2021-11-20T08:52:10.437 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.428+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f88a001a9f0 con 0x7f88a4071c90 2021-11-20T08:52:10.438 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.428+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a410d3b0 con 0x7f88a4071c90 2021-11-20T08:52:10.438 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.428+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a001b890 con 0x7f88a4071c90 2021-11-20T08:52:10.438 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.428+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a410d7c0 con 0x7f88a4071c90 2021-11-20T08:52:10.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.429+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f88a000f070 con 0x7f88a4071c90 2021-11-20T08:52:10.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.429+0000 7f8892ffd700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f888c02fc70 0x7f888c032130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:10.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.429+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f88a0041060 con 0x7f88a4071c90 2021-11-20T08:52:10.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.429+0000 7f88a9ce7700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f888c02fc70 0x7f888c032130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:10.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.429+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a4101d60 con 0x7f88a4071c90 2021-11-20T08:52:10.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.430+0000 7f88a9ce7700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f888c02fc70 0x7f888c032130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f88940096a0 tx=0x7f8894006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:10.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.433+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88a0025460 con 0x7f88a4071c90 2021-11-20T08:52:10.616 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.617+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f88a410e5f0 con 0x7f88a4071c90 2021-11-20T08:52:10.898 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.899+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f88a00257e0 con 0x7f88a4071c90 2021-11-20T08:52:10.899 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.900+0000 7f8892ffd700 1 -- 172.21.15.93:0/1814636184 <== mon.0 v2:172.21.15.93:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f88a0023740 con 0x7f88a4071c90 2021-11-20T08:52:10.902 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f888c02fc70 msgr2=0x7f888c032130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:10.902 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f888c02fc70 0x7f888c032130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f88940096a0 tx=0x7f8894006b40).stop 2021-11-20T08:52:10.903 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 msgr2=0x7f88a410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:10.903 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a410ccc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f88a0006c70 tx=0x7f88a0006ee0).stop 2021-11-20T08:52:10.903 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 shutdown_connections 2021-11-20T08:52:10.903 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f888c02fc70 0x7f888c032130 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:10.903 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 --2- 172.21.15.93:0/1814636184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a4071c90 0x7f88a410ccc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:10.904 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.902+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 >> 172.21.15.93:0/1814636184 conn(0x7f88a40f73e0 msgr2=0x7f88a40f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:10.904 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.903+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 shutdown_connections 2021-11-20T08:52:10.904 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:10.903+0000 7f88ac74c700 1 -- 172.21.15.93:0/1814636184 wait complete. 2021-11-20T08:52:11.210 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:10 smithi093 conmon[33522]: audit 2021-11-20T08:52:10.618283+0000 mon.smithi093 (mon.0) 24 : audit [INF] from='client.? 172.21.15.93:0/1814636184' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-11-20T08:52:11.329 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T08:52:11.813 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.810+0000 7f83cfc63700 1 Processor -- start 2021-11-20T08:52:11.813 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.810+0000 7f83cfc63700 1 -- start start 2021-11-20T08:52:11.814 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.811+0000 7f83cfc63700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c80faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:11.814 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.811+0000 7f83cfc63700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c80fb3e0 con 0x7f83c80faad0 2021-11-20T08:52:11.814 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.811+0000 7f83cd9ff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c80faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:11.814 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.811+0000 7f83cd9ff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c80faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42128/0 (socket says 172.21.15.93:42128) 2021-11-20T08:52:11.815 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.811+0000 7f83cd9ff700 1 -- 172.21.15.93:0/857385231 learned_addr learned my addr 172.21.15.93:0/857385231 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:11.815 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.811+0000 7f83cd9ff700 1 -- 172.21.15.93:0/857385231 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c80fb520 con 0x7f83c80faad0 2021-11-20T08:52:11.815 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.811+0000 7f83cd9ff700 1 --2- 172.21.15.93:0/857385231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c80faef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f83b800b5f0 tx=0x7f83b8006ee0).ready entity=mon.0 client_cookie=596f3f24c7799208 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:11.815 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cc9fd700 1 -- 172.21.15.93:0/857385231 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b800d600 con 0x7f83c80faad0 2021-11-20T08:52:11.816 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cc9fd700 1 -- 172.21.15.93:0/857385231 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f83b800dbd0 con 0x7f83c80faad0 2021-11-20T08:52:11.816 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cc9fd700 1 -- 172.21.15.93:0/857385231 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b8011c00 con 0x7f83c80faad0 2021-11-20T08:52:11.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cfc63700 1 -- 172.21.15.93:0/857385231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 msgr2=0x7f83c80faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:11.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cfc63700 1 --2- 172.21.15.93:0/857385231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c80faef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f83b800b5f0 tx=0x7f83b8006ee0).stop 2021-11-20T08:52:11.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cfc63700 1 -- 172.21.15.93:0/857385231 shutdown_connections 2021-11-20T08:52:11.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cfc63700 1 --2- 172.21.15.93:0/857385231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c80faef0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:11.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cfc63700 1 -- 172.21.15.93:0/857385231 >> 172.21.15.93:0/857385231 conn(0x7f83c80f6310 msgr2=0x7f83c80f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:11.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cfc63700 1 -- 172.21.15.93:0/857385231 shutdown_connections 2021-11-20T08:52:11.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.812+0000 7f83cfc63700 1 -- 172.21.15.93:0/857385231 wait complete. 2021-11-20T08:52:11.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cfc63700 1 Processor -- start 2021-11-20T08:52:11.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cfc63700 1 -- start start 2021-11-20T08:52:11.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cfc63700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c810d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:11.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cfc63700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c806fcb0 con 0x7f83c80faad0 2021-11-20T08:52:11.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cd9ff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c810d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:11.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cd9ff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c810d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42130/0 (socket says 172.21.15.93:42130) 2021-11-20T08:52:11.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cd9ff700 1 -- 172.21.15.93:0/2060323027 learned_addr learned my addr 172.21.15.93:0/2060323027 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:11.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.813+0000 7f83cd9ff700 1 -- 172.21.15.93:0/2060323027 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b800b200 con 0x7f83c80faad0 2021-11-20T08:52:11.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.814+0000 7f83cd9ff700 1 --2- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c810d600 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83b800b1d0 tx=0x7f83b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:11.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.814+0000 7f83b67fc700 1 -- 172.21.15.93:0/2060323027 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b801b410 con 0x7f83c80faad0 2021-11-20T08:52:11.821 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.814+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c806feb0 con 0x7f83c80faad0 2021-11-20T08:52:11.822 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.814+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c810dcb0 con 0x7f83c80faad0 2021-11-20T08:52:11.822 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.814+0000 7f83b67fc700 1 -- 172.21.15.93:0/2060323027 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f83b801b9e0 con 0x7f83c80faad0 2021-11-20T08:52:11.822 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.814+0000 7f83b67fc700 1 -- 172.21.15.93:0/2060323027 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b801a840 con 0x7f83c80faad0 2021-11-20T08:52:11.823 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.815+0000 7f83b67fc700 1 -- 172.21.15.93:0/2060323027 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f83b800f070 con 0x7f83c80faad0 2021-11-20T08:52:11.823 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.815+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c8045ad0 con 0x7f83c80faad0 2021-11-20T08:52:11.823 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.815+0000 7f83b67fc700 1 --2- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f83b00341c0 0x7f83b0036680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:11.823 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.815+0000 7f83cd1fe700 1 -- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f83b00341c0 msgr2=0x7f83b0036680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/1580490298 2021-11-20T08:52:11.824 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.815+0000 7f83b67fc700 1 -- 172.21.15.93:0/2060323027 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f83b8041160 con 0x7f83c80faad0 2021-11-20T08:52:11.824 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.815+0000 7f83cd1fe700 1 --2- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f83b00341c0 0x7f83b0036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T08:52:11.824 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:11.819+0000 7f83b67fc700 1 -- 172.21.15.93:0/2060323027 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83b8022d20 con 0x7f83c80faad0 2021-11-20T08:52:12.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.010+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f83c80fc7d0 con 0x7f83c80faad0 2021-11-20T08:52:12.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.010+0000 7f83b67fc700 1 -- 172.21.15.93:0/2060323027 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f83b8022310 con 0x7f83c80faad0 2021-11-20T08:52:12.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: { 2021-11-20T08:52:12.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 4, 2021-11-20T08:52:12.012 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "available": true, 2021-11-20T08:52:12.012 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "active_name": "smithi093.qdikkm", 2021-11-20T08:52:12.013 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T08:52:12.014 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: } 2021-11-20T08:52:12.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f83b00341c0 msgr2=0x7f83b0036680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T08:52:12.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 --2- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f83b00341c0 0x7f83b0036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:12.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 msgr2=0x7f83c810d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:12.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 --2- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c810d600 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83b800b1d0 tx=0x7f83b8006ee0).stop 2021-11-20T08:52:12.018 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 shutdown_connections 2021-11-20T08:52:12.018 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 --2- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f83b00341c0 0x7f83b0036680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:12.018 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 --2- 172.21.15.93:0/2060323027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f83c80faad0 0x7f83c810d600 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:12.018 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.012+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 >> 172.21.15.93:0/2060323027 conn(0x7f83c80f6310 msgr2=0x7f83c80f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:12.019 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.013+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 shutdown_connections 2021-11-20T08:52:12.019 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.013+0000 7f83cfc63700 1 -- 172.21.15.93:0/2060323027 wait complete. 2021-11-20T08:52:12.154 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:11 smithi093 conmon[33522]: audit 2021-11-20T08:52:10 2021-11-20T08:52:12.155 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:11 smithi093 conmon[33522]: .901442+0000 mon.smithi093 (mon.0) 25 : audit [INF] from='client.? 172.21.15.93:0/1814636184' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-11-20T08:52:12.155 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:11 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:52:12.155 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:11 smithi093 conmon[33522]: 08:52:10.901498+0000 mon.smithi093 (mon.0) 26 : cluster 2021-11-20T08:52:12.156 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:11 smithi093 conmon[33522]: [DBG] mgrmap e4: smithi093.qdikkm(active, since 3s) 2021-11-20T08:52:12.366 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for the mgr to restart... 2021-11-20T08:52:12.366 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for mgr epoch 4... 2021-11-20T08:52:12.366 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T08:52:12.829 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.829+0000 7f2420f4f700 1 Processor -- start 2021-11-20T08:52:12.830 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.829+0000 7f2420f4f700 1 -- start start 2021-11-20T08:52:12.830 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.829+0000 7f2420f4f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c100c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:12.830 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.829+0000 7f2420f4f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0fc680 con 0x7f241c06fc70 2021-11-20T08:52:12.831 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.829+0000 7f241a59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:12.831 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.830+0000 7f241a59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42136/0 (socket says 172.21.15.93:42136) 2021-11-20T08:52:12.831 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.830+0000 7f241a59c700 1 -- 172.21.15.93:0/2074500889 learned_addr learned my addr 172.21.15.93:0/2074500889 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:12.831 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.830+0000 7f241a59c700 1 -- 172.21.15.93:0/2074500889 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c1010f0 con 0x7f241c06fc70 2021-11-20T08:52:12.832 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.830+0000 7f241a59c700 1 --2- 172.21.15.93:0/2074500889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c100c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f241000b5f0 tx=0x7f2410006ee0).ready entity=mon.0 client_cookie=e2aecdcb79da59fb server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:12.832 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.830+0000 7f241959a700 1 -- 172.21.15.93:0/2074500889 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f241000d600 con 0x7f241c06fc70 2021-11-20T08:52:12.832 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.830+0000 7f241959a700 1 -- 172.21.15.93:0/2074500889 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f241000dbd0 con 0x7f241c06fc70 2021-11-20T08:52:12.833 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.830+0000 7f241959a700 1 -- 172.21.15.93:0/2074500889 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2410011c00 con 0x7f241c06fc70 2021-11-20T08:52:12.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.831+0000 7f2420f4f700 1 -- 172.21.15.93:0/2074500889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 msgr2=0x7f241c100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:12.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.831+0000 7f2420f4f700 1 --2- 172.21.15.93:0/2074500889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c100c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f241000b5f0 tx=0x7f2410006ee0).stop 2021-11-20T08:52:12.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.831+0000 7f2420f4f700 1 -- 172.21.15.93:0/2074500889 shutdown_connections 2021-11-20T08:52:12.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.831+0000 7f2420f4f700 1 --2- 172.21.15.93:0/2074500889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c100c00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:12.836 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.831+0000 7f2420f4f700 1 -- 172.21.15.93:0/2074500889 >> 172.21.15.93:0/2074500889 conn(0x7f241c0f6280 msgr2=0x7f241c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:12.836 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.831+0000 7f2420f4f700 1 -- 172.21.15.93:0/2074500889 shutdown_connections 2021-11-20T08:52:12.836 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.831+0000 7f2420f4f700 1 -- 172.21.15.93:0/2074500889 wait complete. 2021-11-20T08:52:12.837 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f2420f4f700 1 Processor -- start 2021-11-20T08:52:12.837 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f2420f4f700 1 -- start start 2021-11-20T08:52:12.837 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f2420f4f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c10cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:12.837 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f2420f4f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c10d0d0 con 0x7f241c06fc70 2021-11-20T08:52:12.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f241a59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c10cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:12.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f241a59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c10cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42138/0 (socket says 172.21.15.93:42138) 2021-11-20T08:52:12.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f241a59c700 1 -- 172.21.15.93:0/1396101774 learned_addr learned my addr 172.21.15.93:0/1396101774 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:12.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f241a59c700 1 -- 172.21.15.93:0/1396101774 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241000b200 con 0x7f241c06fc70 2021-11-20T08:52:12.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f241a59c700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c10cbe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f2410000c00 tx=0x7f2410006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:12.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.832+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2410011690 con 0x7f241c06fc70 2021-11-20T08:52:12.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.833+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c10d2d0 con 0x7f241c06fc70 2021-11-20T08:52:12.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.833+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c10d6e0 con 0x7f241c06fc70 2021-11-20T08:52:12.840 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.833+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f241000d9f0 con 0x7f241c06fc70 2021-11-20T08:52:12.840 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.833+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f241001b710 con 0x7f241c06fc70 2021-11-20T08:52:12.840 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.834+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f241000f070 con 0x7f241c06fc70 2021-11-20T08:52:12.840 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.834+0000 7f240affd700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:12.840 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.834+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 --> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2400032930 con 0x7f240002fdb0 2021-11-20T08:52:12.841 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.834+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2410041000 con 0x7f241c06fc70 2021-11-20T08:52:12.841 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.834+0000 7f2419d9b700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 msgr2=0x7f2400032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/1580490298 2021-11-20T08:52:12.841 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:12.834+0000 7f2419d9b700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T08:52:13.034 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:13.035+0000 7f2419d9b700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 msgr2=0x7f2400032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/1580490298 2021-11-20T08:52:13.035 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:13.035+0000 7f2419d9b700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T08:52:13.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:12 smithi093 conmon[33522]: audit 2021-11-20T08:52:12.011532+0000 mon.smithi093 (mon 2021-11-20T08:52:13.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:12 smithi093 conmon[33522]: .0) 27 : audit [DBG] from='client.? 172.21.15.93:0/2060323027' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T08:52:13.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:13.436+0000 7f2419d9b700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 msgr2=0x7f2400032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/1580490298 2021-11-20T08:52:13.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:13.436+0000 7f2419d9b700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T08:52:14.236 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:14.237+0000 7f2419d9b700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 msgr2=0x7f2400032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/1580490298 2021-11-20T08:52:14.236 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:14.237+0000 7f2419d9b700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T08:52:15.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:15.838+0000 7f2419d9b700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 msgr2=0x7f2400032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/1580490298 2021-11-20T08:52:15.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:15.838+0000 7f2419d9b700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T08:52:15.898 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:15.898+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f241001aaf0 con 0x7f241c06fc70 2021-11-20T08:52:15.899 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:15.898+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 msgr2=0x7f2400032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T08:52:15.900 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:15.898+0000 7f240affd700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:16.899 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.900+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f2410042750 con 0x7f241c06fc70 2021-11-20T08:52:16.900 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.900+0000 7f240affd700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f2400033880 0x7f2400035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:16.900 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.900+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2400032930 con 0x7f2400033880 2021-11-20T08:52:16.901 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.900+0000 7f2419d9b700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f2400033880 0x7f2400035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:16.901 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.901+0000 7f2419d9b700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f2400033880 0x7f2400035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f240c006fd0 tx=0x7f240c006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:16.901 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.902+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f2400032930 con 0x7f2400033880 2021-11-20T08:52:16.907 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.908+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f23f8000dc0 con 0x7f2400033880 2021-11-20T08:52:16.909 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.908+0000 7f240affd700 1 -- 172.21.15.93:0/1396101774 <== mgr.14114 v2:172.21.15.93:6800/3748247363 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f23f8000dc0 con 0x7f2400033880 2021-11-20T08:52:16.909 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: { 2021-11-20T08:52:16.910 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2021-11-20T08:52:16.910 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T08:52:16.910 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: } 2021-11-20T08:52:16.910 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.909+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f2400033880 msgr2=0x7f2400035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:16.911 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.909+0000 7f2420f4f700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f2400033880 0x7f2400035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f240c006fd0 tx=0x7f240c006d70).stop 2021-11-20T08:52:16.911 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.909+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 msgr2=0x7f241c10cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:16.911 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.909+0000 7f2420f4f700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c10cbe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f2410000c00 tx=0x7f2410006ee0).stop 2021-11-20T08:52:16.911 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.910+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 shutdown_connections 2021-11-20T08:52:16.912 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.910+0000 7f2420f4f700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f241c06fc70 0x7f241c10cbe0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:16.912 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.910+0000 7f2420f4f700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f2400033880 0x7f2400035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:16.912 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.910+0000 7f2420f4f700 1 --2- 172.21.15.93:0/1396101774 >> [v2:172.21.15.93:6800/1580490298,v1:172.21.15.93:6801/1580490298] conn(0x7f240002fdb0 0x7f2400032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:16.912 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.910+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 >> 172.21.15.93:0/1396101774 conn(0x7f241c0f6280 msgr2=0x7f241c0f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:16.913 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.910+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 shutdown_connections 2021-11-20T08:52:16.913 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:16.910+0000 7f2420f4f700 1 -- 172.21.15.93:0/1396101774 wait complete. 2021-11-20T08:52:17.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: cluster 2021-11-20T08:52:15.847522+0000 mon.smithi093 (mon.0) 28 : cluster [INF] Active manager daemon smithi093.qdikkm restarted 2021-11-20T08:52:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: cluster 2021-11-20T08:52:15.848020+0000 mon.smithi093 (mon.0) 29 : cluster [INF] Activating manager daemon smithi093.qdikkm 2021-11-20T08:52:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: cluster 2021-11-20T08:52:15.849657+0000 mon.smithi093 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-11-20T08:52:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: cluster 2021-11-20T08:52:15.900817+0000 mon.smithi093 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi093.qdikkm(active, starting, since 0.0528942s) 2021-11-20T08:52:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15.901753+0000 mon.smithi093 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi093"}]: dispatch 2021-11-20T08:52:17.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15.902315+0000 mon.smithi093 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr metadata", "who": "smithi093.qdikkm", "id": "smithi093.qdikkm"}]: dispatch 2021-11-20T08:52:17.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15.902921+0000 mon.smithi093 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T08:52:17.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15.903094+0000 mon.smithi093 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T08:52:17.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15.903229+0000 mon.smithi093 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T08:52:17.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: cluster 2021-11-20T08:52:15.917330+0000 mon.smithi093 (mon.0) 37 : cluster [INF] Manager daemon smithi093.qdikkm is now available 2021-11-20T08:52:17.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15.933796+0000 mon.smithi093 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:17.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15.939239+0000 mon.smithi093 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:17.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:15 2021-11-20T08:52:17.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: .944527+0000 mon.smithi093 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:17.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:16.034364+0000 mon.smithi093 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:17.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:52:17.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 52:16.035044+0000 mon.smithi093 (mon 2021-11-20T08:52:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: .0) 42 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 2021-11-20T08:52:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 2021-11-20T08:52: 2021-11-20T08:52:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 16.036455+0000 mon.smithi093 (mon. 2021-11-20T08:52:17.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 0) 43 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:17.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 2021-11-20T08:52:17.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20T08:52:17.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 2021-11-20T08:52:16 2021-11-20T08:52:17.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: .037880+0000 mon.smithi093 (mon.0) 2021-11-20T08:52:17.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 44 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:17.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:52:17.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: -20T08:52:16.047753+0000 mon.smithi093 2021-11-20T08:52:17.220 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: (mon.0) 45 : audit [INF] 2021-11-20T08:52:17.220 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:52:17.220 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: audit 2021-11-20 2021-11-20T08:52:17.220 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: T08:52:16.048867+0000 mon.smithi093 (mon. 2021-11-20T08:52:17.221 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:16 smithi093 conmon[33522]: 0) 46 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:52:18.161 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:17 smithi093 conmon[33522]: audit 2021-11-20T08:52:16.902792+0000 mgr.smithi093.qdikkm (mgr.14114) 1 : audit 2021-11-20T08:52:18.162 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:17 smithi093 conmon[33522]: [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T08:52:18.162 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:17 smithi093 conmon[33522]: cluster 2021-11-20T08:52:16.903294+0000 mon.smithi093 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi093.qdikkm(active, since 1.05537s) 2021-11-20T08:52:18.162 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:17 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:52:18.162 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:17 smithi093 conmon[33522]: 20T08:52:16.909627+0000 mgr.smithi093.qdikkm (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T08:52:18.718 INFO:teuthology.orchestra.run.smithi093.stderr:mgr epoch 4 is available 2021-11-20T08:52:18.719 INFO:teuthology.orchestra.run.smithi093.stderr:Setting orchestrator backend to cephadm... 2021-11-20T08:52:18.719 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2021-11-20T08:52:19.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:18 smithi093 conmon[33522]: cluster 2021-11-20T08:52:17.908215+0000 mon.smithi093 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi093.qdikkm(active, since 2s) 2021-11-20T08:52:20.586 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.517+0000 7fb1415f1700 1 Processor -- start 2021-11-20T08:52:20.586 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.517+0000 7fb1415f1700 1 -- start start 2021-11-20T08:52:20.586 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb1415f1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c0fc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:20.587 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb1415f1700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0fc9a0 con 0x7fb13c0fc090 2021-11-20T08:52:20.587 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb13affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c0fc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:20.587 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb13affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c0fc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42174/0 (socket says 172.21.15.93:42174) 2021-11-20T08:52:20.587 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb13affd700 1 -- 172.21.15.93:0/4212530577 learned_addr learned my addr 172.21.15.93:0/4212530577 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:20.588 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb13affd700 1 -- 172.21.15.93:0/4212530577 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0fcae0 con 0x7fb13c0fc090 2021-11-20T08:52:20.588 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb13affd700 1 --2- 172.21.15.93:0/4212530577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c0fc4b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb12c00b870 tx=0x7fb12c006f70).ready entity=mon.0 client_cookie=b48783a1078a35e8 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:20.588 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb139ffb700 1 -- 172.21.15.93:0/4212530577 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c00da20 con 0x7fb13c0fc090 2021-11-20T08:52:20.588 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb139ffb700 1 -- 172.21.15.93:0/4212530577 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb12c010470 con 0x7fb13c0fc090 2021-11-20T08:52:20.589 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.518+0000 7fb139ffb700 1 -- 172.21.15.93:0/4212530577 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c00f5b0 con 0x7fb13c0fc090 2021-11-20T08:52:20.589 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.519+0000 7fb1415f1700 1 -- 172.21.15.93:0/4212530577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 msgr2=0x7fb13c0fc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:20.589 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.519+0000 7fb1415f1700 1 --2- 172.21.15.93:0/4212530577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c0fc4b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb12c00b870 tx=0x7fb12c006f70).stop 2021-11-20T08:52:20.589 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.519+0000 7fb1415f1700 1 -- 172.21.15.93:0/4212530577 shutdown_connections 2021-11-20T08:52:20.589 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.519+0000 7fb1415f1700 1 --2- 172.21.15.93:0/4212530577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c0fc4b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:20.590 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.519+0000 7fb1415f1700 1 -- 172.21.15.93:0/4212530577 >> 172.21.15.93:0/4212530577 conn(0x7fb13c0f5a80 msgr2=0x7fb13c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:20.590 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.519+0000 7fb1415f1700 1 -- 172.21.15.93:0/4212530577 shutdown_connections 2021-11-20T08:52:20.590 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.519+0000 7fb1415f1700 1 -- 172.21.15.93:0/4212530577 wait complete. 2021-11-20T08:52:20.590 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb1415f1700 1 Processor -- start 2021-11-20T08:52:20.591 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb1415f1700 1 -- start start 2021-11-20T08:52:20.591 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb1415f1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c108750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:20.591 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb1415f1700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c108c40 con 0x7fb13c0fc090 2021-11-20T08:52:20.591 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb13affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:20.592 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb13affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42176/0 (socket says 172.21.15.93:42176) 2021-11-20T08:52:20.592 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb13affd700 1 -- 172.21.15.93:0/1243083120 learned_addr learned my addr 172.21.15.93:0/1243083120 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:20.592 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb13affd700 1 -- 172.21.15.93:0/1243083120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c00b480 con 0x7fb13c0fc090 2021-11-20T08:52:20.592 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.520+0000 7fb13affd700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c108750 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fb12c006b80 tx=0x7fb12c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:20.592 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.521+0000 7fb1237fe700 1 -- 172.21.15.93:0/1243083120 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c00f3d0 con 0x7fb13c0fc090 2021-11-20T08:52:20.593 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.521+0000 7fb1237fe700 1 -- 172.21.15.93:0/1243083120 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb12c00fef0 con 0x7fb13c0fc090 2021-11-20T08:52:20.593 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.521+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c108e40 con 0x7fb13c0fc090 2021-11-20T08:52:20.593 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.521+0000 7fb1237fe700 1 -- 172.21.15.93:0/1243083120 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c019c70 con 0x7fb13c0fc090 2021-11-20T08:52:20.594 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.521+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c109250 con 0x7fb13c0fc090 2021-11-20T08:52:20.594 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.521+0000 7fb1237fe700 1 -- 172.21.15.93:0/1243083120 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb12c016030 con 0x7fb13c0fc090 2021-11-20T08:52:20.594 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.522+0000 7fb1237fe700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12802fd50 0x7fb128032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:20.594 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.522+0000 7fb1237fe700 1 -- 172.21.15.93:0/1243083120 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb12c043ea0 con 0x7fb13c0fc090 2021-11-20T08:52:20.595 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.522+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c045ad0 con 0x7fb13c0fc090 2021-11-20T08:52:20.595 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.522+0000 7fb13a7fc700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12802fd50 0x7fb128032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:20.595 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.522+0000 7fb13a7fc700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12802fd50 0x7fb128032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb1300096a0 tx=0x7fb130006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:20.595 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.526+0000 7fb1237fe700 1 -- 172.21.15.93:0/1243083120 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb12c06a2b0 con 0x7fb13c0fc090 2021-11-20T08:52:20.708 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.709+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb13c109b30 con 0x7fb12802fd50 2021-11-20T08:52:20.713 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.714+0000 7fb1237fe700 1 -- 172.21.15.93:0/1243083120 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb13c109b30 con 0x7fb12802fd50 2021-11-20T08:52:20.716 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12802fd50 msgr2=0x7fb128032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:20.716 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12802fd50 0x7fb128032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb1300096a0 tx=0x7fb130006b40).stop 2021-11-20T08:52:20.716 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 msgr2=0x7fb13c108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:20.717 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c108750 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fb12c006b80 tx=0x7fb12c006f70).stop 2021-11-20T08:52:20.717 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 shutdown_connections 2021-11-20T08:52:20.717 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12802fd50 0x7fb128032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:20.717 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 --2- 172.21.15.93:0/1243083120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb13c0fc090 0x7fb13c108750 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:20.718 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.716+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 >> 172.21.15.93:0/1243083120 conn(0x7fb13c0f5a80 msgr2=0x7fb13c0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:20.718 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.717+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 shutdown_connections 2021-11-20T08:52:20.718 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:20.717+0000 7fb1415f1700 1 -- 172.21.15.93:0/1243083120 wait complete. 2021-11-20T08:52:21.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: audit 2021-11-20T08:52:20.711157+0000 mgr.smithi093.qdikkm (mgr.14114) 3 : audit 2021-11-20T08:52:21.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:21.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: audit 2021-11-20T08:52:20.715126+0000 mon.smithi093 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:21.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: 2021-11-20T08:52:21.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:52:21.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: 11-20T08:52:20.718148 2021-11-20T08:52:21.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 50 : 2021-11-20T08:52:21.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:21 smithi093 conmon[33522]: audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:22.364 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2021-11-20T08:52:24.051 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.045+0000 7fbaa4ee2700 1 Processor -- start 2021-11-20T08:52:24.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.045+0000 7fbaa4ee2700 1 -- start start 2021-11-20T08:52:24.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.045+0000 7fbaa4ee2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa00faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:24.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fbaa4ee2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00fb480 con 0x7fbaa00faae0 2021-11-20T08:52:24.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa00faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:24.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa00faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42180/0 (socket says 172.21.15.93:42180) 2021-11-20T08:52:24.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9e59c700 1 -- 172.21.15.93:0/873622401 learned_addr learned my addr 172.21.15.93:0/873622401 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:24.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9e59c700 1 -- 172.21.15.93:0/873622401 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00fb5c0 con 0x7fbaa00faae0 2021-11-20T08:52:24.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9e59c700 1 --2- 172.21.15.93:0/873622401 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa00faf00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fba9000b8d0 tx=0x7fba90006f70).ready entity=mon.0 client_cookie=416ed1c31f60b515 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:24.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9d59a700 1 -- 172.21.15.93:0/873622401 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba9000d940 con 0x7fbaa00faae0 2021-11-20T08:52:24.055 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9d59a700 1 -- 172.21.15.93:0/873622401 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fba90012770 con 0x7fbaa00faae0 2021-11-20T08:52:24.056 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.046+0000 7fba9d59a700 1 -- 172.21.15.93:0/873622401 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba9001a620 con 0x7fbaa00faae0 2021-11-20T08:52:24.056 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.047+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/873622401 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 msgr2=0x7fbaa00faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:24.056 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.047+0000 7fbaa4ee2700 1 --2- 172.21.15.93:0/873622401 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa00faf00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fba9000b8d0 tx=0x7fba90006f70).stop 2021-11-20T08:52:24.056 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.047+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/873622401 shutdown_connections 2021-11-20T08:52:24.057 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.047+0000 7fbaa4ee2700 1 --2- 172.21.15.93:0/873622401 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa00faf00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:24.057 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.047+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/873622401 >> 172.21.15.93:0/873622401 conn(0x7fbaa00f6280 msgr2=0x7fbaa00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:24.057 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.047+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/873622401 shutdown_connections 2021-11-20T08:52:24.057 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.047+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/873622401 wait complete. 2021-11-20T08:52:24.058 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fbaa4ee2700 1 Processor -- start 2021-11-20T08:52:24.058 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fbaa4ee2700 1 -- start start 2021-11-20T08:52:24.058 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fbaa4ee2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa0108750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:24.058 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fbaa4ee2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0108c40 con 0x7fbaa00faae0 2021-11-20T08:52:24.059 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fba9e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa0108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:24.059 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fba9e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa0108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42182/0 (socket says 172.21.15.93:42182) 2021-11-20T08:52:24.059 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fba9e59c700 1 -- 172.21.15.93:0/3258994462 learned_addr learned my addr 172.21.15.93:0/3258994462 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:24.059 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fba9e59c700 1 -- 172.21.15.93:0/3258994462 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9000b480 con 0x7fbaa00faae0 2021-11-20T08:52:24.059 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.048+0000 7fba9e59c700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa0108750 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fba9000d5d0 tx=0x7fba90006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:24.060 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.049+0000 7fba96ffd700 1 -- 172.21.15.93:0/3258994462 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba9001a440 con 0x7fbaa00faae0 2021-11-20T08:52:24.060 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.049+0000 7fba96ffd700 1 -- 172.21.15.93:0/3258994462 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fba9001b450 con 0x7fbaa00faae0 2021-11-20T08:52:24.060 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.049+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa0108e40 con 0x7fbaa00faae0 2021-11-20T08:52:24.060 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.049+0000 7fba96ffd700 1 -- 172.21.15.93:0/3258994462 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90024620 con 0x7fbaa00faae0 2021-11-20T08:52:24.061 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.049+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa0109250 con 0x7fbaa00faae0 2021-11-20T08:52:24.061 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.050+0000 7fba96ffd700 1 -- 172.21.15.93:0/3258994462 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fba90018030 con 0x7fbaa00faae0 2021-11-20T08:52:24.061 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.050+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa0045ad0 con 0x7fbaa00faae0 2021-11-20T08:52:24.061 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.050+0000 7fba96ffd700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fba8002fdb0 0x7fba80032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:24.062 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.050+0000 7fba96ffd700 1 -- 172.21.15.93:0/3258994462 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fba900458a0 con 0x7fbaa00faae0 2021-11-20T08:52:24.062 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.050+0000 7fba9dd9b700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fba8002fdb0 0x7fba80032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:24.062 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.050+0000 7fba9dd9b700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fba8002fdb0 0x7fba80032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:24.063 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.054+0000 7fba96ffd700 1 -- 172.21.15.93:0/3258994462 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba90022570 con 0x7fbaa00faae0 2021-11-20T08:52:24.205 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.205+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaa0109b90 con 0x7fba8002fdb0 2021-11-20T08:52:24.205 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.206+0000 7fba96ffd700 1 -- 172.21.15.93:0/3258994462 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fbaa0109b90 con 0x7fba8002fdb0 2021-11-20T08:52:24.206 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: value unchanged 2021-11-20T08:52:24.209 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fba8002fdb0 msgr2=0x7fba80032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:24.210 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fba8002fdb0 0x7fba80032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).stop 2021-11-20T08:52:24.210 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 msgr2=0x7fbaa0108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:24.211 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa0108750 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fba9000d5d0 tx=0x7fba90006f70).stop 2021-11-20T08:52:24.211 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 shutdown_connections 2021-11-20T08:52:24.211 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fba8002fdb0 0x7fba80032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:24.212 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 --2- 172.21.15.93:0/3258994462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbaa00faae0 0x7fbaa0108750 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:24.212 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 >> 172.21.15.93:0/3258994462 conn(0x7fbaa00f6280 msgr2=0x7fbaa00ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:24.212 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 shutdown_connections 2021-11-20T08:52:24.212 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.208+0000 7fbaa4ee2700 1 -- 172.21.15.93:0/3258994462 wait complete. 2021-11-20T08:52:24.509 INFO:teuthology.orchestra.run.smithi093.stderr:Generating ssh key... 2021-11-20T08:52:24.509 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2021-11-20T08:52:24.983 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.983+0000 7f098910b700 1 Processor -- start 2021-11-20T08:52:24.984 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.983+0000 7f098910b700 1 -- start start 2021-11-20T08:52:24.984 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.983+0000 7f098910b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f09840fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:24.984 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.983+0000 7f098910b700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09840fc4e0 con 0x7f09840fbbd0 2021-11-20T08:52:24.984 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.983+0000 7f0982d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f09840fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:24.984 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.983+0000 7f0982d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f09840fbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42186/0 (socket says 172.21.15.93:42186) 2021-11-20T08:52:24.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.983+0000 7f0982d9d700 1 -- 172.21.15.93:0/126493415 learned_addr learned my addr 172.21.15.93:0/126493415 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:24.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.984+0000 7f0982d9d700 1 -- 172.21.15.93:0/126493415 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09840fc620 con 0x7f09840fbbd0 2021-11-20T08:52:24.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.984+0000 7f0982d9d700 1 --2- 172.21.15.93:0/126493415 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f09840fbff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0974009960 tx=0x7f09740092d0).ready entity=mon.0 client_cookie=91534aaaa14b3565 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:24.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.984+0000 7f0981d9b700 1 -- 172.21.15.93:0/126493415 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f097400b9c0 con 0x7f09840fbbd0 2021-11-20T08:52:24.988 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.984+0000 7f0981d9b700 1 -- 172.21.15.93:0/126493415 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0974011400 con 0x7f09840fbbd0 2021-11-20T08:52:24.988 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.984+0000 7f0981d9b700 1 -- 172.21.15.93:0/126493415 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09740103a0 con 0x7f09840fbbd0 2021-11-20T08:52:24.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.984+0000 7f098910b700 1 -- 172.21.15.93:0/126493415 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 msgr2=0x7f09840fbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:24.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.984+0000 7f098910b700 1 --2- 172.21.15.93:0/126493415 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f09840fbff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0974009960 tx=0x7f09740092d0).stop 2021-11-20T08:52:24.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.985+0000 7f098910b700 1 -- 172.21.15.93:0/126493415 shutdown_connections 2021-11-20T08:52:24.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.985+0000 7f098910b700 1 --2- 172.21.15.93:0/126493415 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f09840fbff0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:24.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.985+0000 7f098910b700 1 -- 172.21.15.93:0/126493415 >> 172.21.15.93:0/126493415 conn(0x7f09840f73f0 msgr2=0x7f09840f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:24.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.985+0000 7f098910b700 1 -- 172.21.15.93:0/126493415 shutdown_connections 2021-11-20T08:52:24.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.985+0000 7f098910b700 1 -- 172.21.15.93:0/126493415 wait complete. 2021-11-20T08:52:24.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.985+0000 7f098910b700 1 Processor -- start 2021-11-20T08:52:24.991 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.985+0000 7f098910b700 1 -- start start 2021-11-20T08:52:24.991 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f098910b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f0984071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:24.991 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f098910b700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09840701f0 con 0x7f09840fbbd0 2021-11-20T08:52:24.992 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f0982d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f0984071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:24.992 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f0982d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f0984071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42188/0 (socket says 172.21.15.93:42188) 2021-11-20T08:52:24.992 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f0982d9d700 1 -- 172.21.15.93:0/660110261 learned_addr learned my addr 172.21.15.93:0/660110261 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:24.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f0982d9d700 1 -- 172.21.15.93:0/660110261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0974009570 con 0x7f09840fbbd0 2021-11-20T08:52:24.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f0982d9d700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f0984071d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f097400ba90 tx=0x7f09740092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:24.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f097b7fe700 1 -- 172.21.15.93:0/660110261 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0974010870 con 0x7f09840fbbd0 2021-11-20T08:52:24.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f097b7fe700 1 -- 172.21.15.93:0/660110261 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0974010e40 con 0x7f09840fbbd0 2021-11-20T08:52:24.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f097b7fe700 1 -- 172.21.15.93:0/660110261 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f097401ae10 con 0x7f09840fbbd0 2021-11-20T08:52:24.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0984070390 con 0x7f09840fbbd0 2021-11-20T08:52:24.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.986+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09840707a0 con 0x7f09840fbbd0 2021-11-20T08:52:24.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.987+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0984045ad0 con 0x7f09840fbbd0 2021-11-20T08:52:24.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.988+0000 7f097b7fe700 1 -- 172.21.15.93:0/660110261 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f097402a070 con 0x7f09840fbbd0 2021-11-20T08:52:24.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.988+0000 7f097b7fe700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f096402fd00 0x7f09640321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:24.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.988+0000 7f097b7fe700 1 -- 172.21.15.93:0/660110261 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0974041a90 con 0x7f09840fbbd0 2021-11-20T08:52:24.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.988+0000 7f098259c700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f096402fd00 0x7f09640321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:24.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.988+0000 7f098259c700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f096402fd00 0x7f09640321c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f096c0096a0 tx=0x7f096c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:24.996 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:24.994+0000 7f097b7fe700 1 -- 172.21.15.93:0/660110261 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09740109d0 con 0x7f09840fbbd0 2021-11-20T08:52:25.142 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.143+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f09840fda50 con 0x7f096402fd00 2021-11-20T08:52:25.244 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:24 smithi093 conmon[33522]: audit 2021-11-20T08:52:24.207038+0000 mgr.smithi093.qdikkm (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:25.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.255+0000 7f097b7fe700 1 -- 172.21.15.93:0/660110261 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f09840fda50 con 0x7f096402fd00 2021-11-20T08:52:25.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f096402fd00 msgr2=0x7f09640321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:25.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f096402fd00 0x7f09640321c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f096c0096a0 tx=0x7f096c006b40).stop 2021-11-20T08:52:25.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 msgr2=0x7f0984071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:25.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f0984071d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f097400ba90 tx=0x7f09740092d0).stop 2021-11-20T08:52:25.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 shutdown_connections 2021-11-20T08:52:25.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f096402fd00 0x7f09640321c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:25.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 --2- 172.21.15.93:0/660110261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09840fbbd0 0x7f0984071d70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:25.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 >> 172.21.15.93:0/660110261 conn(0x7f09840f73f0 msgr2=0x7f09840f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:25.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 shutdown_connections 2021-11-20T08:52:25.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:25.256+0000 7f098910b700 1 -- 172.21.15.93:0/660110261 wait complete. 2021-11-20T08:52:25.544 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2021-11-20T08:52:26.010 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.008+0000 7f7908be0700 1 Processor -- start 2021-11-20T08:52:26.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.008+0000 7f7908be0700 1 -- start start 2021-11-20T08:52:26.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.008+0000 7f7908be0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f79040fe830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:26.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.008+0000 7f7908be0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79040fed20 con 0x7f790406fc30 2021-11-20T08:52:26.011 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.008+0000 7f790259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f79040fe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:26.012 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.008+0000 7f790259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f79040fe830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42192/0 (socket says 172.21.15.93:42192) 2021-11-20T08:52:26.012 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.008+0000 7f790259c700 1 -- 172.21.15.93:0/118470412 learned_addr learned my addr 172.21.15.93:0/118470412 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:26.012 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f790259c700 1 -- 172.21.15.93:0/118470412 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040fee60 con 0x7f790406fc30 2021-11-20T08:52:26.012 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f790259c700 1 --2- 172.21.15.93:0/118470412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f79040fe830 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f78f400b5f0 tx=0x7f78f4006ee0).ready entity=mon.0 client_cookie=ab8b48505c46d5ec server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:26.013 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7901d9b700 1 -- 172.21.15.93:0/118470412 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f400d600 con 0x7f790406fc30 2021-11-20T08:52:26.013 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7901d9b700 1 -- 172.21.15.93:0/118470412 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f78f400dbd0 con 0x7f790406fc30 2021-11-20T08:52:26.013 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7901d9b700 1 -- 172.21.15.93:0/118470412 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f4011c00 con 0x7f790406fc30 2021-11-20T08:52:26.013 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7908be0700 1 -- 172.21.15.93:0/118470412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 msgr2=0x7f79040fe830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:26.013 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7908be0700 1 --2- 172.21.15.93:0/118470412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f79040fe830 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f78f400b5f0 tx=0x7f78f4006ee0).stop 2021-11-20T08:52:26.014 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7908be0700 1 -- 172.21.15.93:0/118470412 shutdown_connections 2021-11-20T08:52:26.014 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7908be0700 1 --2- 172.21.15.93:0/118470412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f79040fe830 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:26.014 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.009+0000 7f7908be0700 1 -- 172.21.15.93:0/118470412 >> 172.21.15.93:0/118470412 conn(0x7f79040f6310 msgr2=0x7f79040f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:26.014 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.010+0000 7f7908be0700 1 -- 172.21.15.93:0/118470412 shutdown_connections 2021-11-20T08:52:26.015 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.010+0000 7f7908be0700 1 -- 172.21.15.93:0/118470412 wait complete. 2021-11-20T08:52:26.015 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.010+0000 7f7908be0700 1 Processor -- start 2021-11-20T08:52:26.015 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.010+0000 7f7908be0700 1 -- start start 2021-11-20T08:52:26.015 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f7908be0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f790410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:26.016 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f7908be0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790410aef0 con 0x7f790406fc30 2021-11-20T08:52:26.016 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f790259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f790410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:26.016 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f790259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f790410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42194/0 (socket says 172.21.15.93:42194) 2021-11-20T08:52:26.016 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f790259c700 1 -- 172.21.15.93:0/2831731396 learned_addr learned my addr 172.21.15.93:0/2831731396 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:26.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f790259c700 1 -- 172.21.15.93:0/2831731396 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f400b200 con 0x7f790406fc30 2021-11-20T08:52:26.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f790259c700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f790410aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f78f400b1d0 tx=0x7f78f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:26.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f78fb7fe700 1 -- 172.21.15.93:0/2831731396 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f4011a20 con 0x7f790406fc30 2021-11-20T08:52:26.017 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f78fb7fe700 1 -- 172.21.15.93:0/2831731396 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f78f401b880 con 0x7f790406fc30 2021-11-20T08:52:26.018 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790410b0f0 con 0x7f790406fc30 2021-11-20T08:52:26.019 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f78fb7fe700 1 -- 172.21.15.93:0/2831731396 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f401a700 con 0x7f790406fc30 2021-11-20T08:52:26.019 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.011+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790410b500 con 0x7f790406fc30 2021-11-20T08:52:26.019 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.012+0000 7f78fb7fe700 1 -- 172.21.15.93:0/2831731396 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f78f400f070 con 0x7f790406fc30 2021-11-20T08:52:26.020 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.012+0000 7f78fb7fe700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f78e402fd50 0x7f78e4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:26.020 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.012+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904045ad0 con 0x7f790406fc30 2021-11-20T08:52:26.020 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.013+0000 7f78fb7fe700 1 -- 172.21.15.93:0/2831731396 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f78f40410c0 con 0x7f790406fc30 2021-11-20T08:52:26.020 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.013+0000 7f78f9d9b700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f78e402fd50 0x7f78e4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:26.020 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.013+0000 7f78f9d9b700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f78e402fd50 0x7f78e4032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f78ec0096a0 tx=0x7f78ec006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:26.021 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.017+0000 7f78fb7fe700 1 -- 172.21.15.93:0/2831731396 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78f4022af0 con 0x7f790406fc30 2021-11-20T08:52:26.168 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.169+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f790410be40 con 0x7f78e402fd50 2021-11-20T08:52:26.168 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.169+0000 7f78fb7fe700 1 -- 172.21.15.93:0/2831731396 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f790410be40 con 0x7f78e402fd50 2021-11-20T08:52:26.169 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj3OnmdS6uXgb20MvhfkMc1OUojWrpYTu+JfuMhv50/XZ8QkN8u5yQsEDN4j8nMaxhO/1+uGS9TT+Zf3bKjKnYfYD/5sy8tbFU1hqHUese/3ifHfl7nPuEzxAq/ITrT0dfsZpjDdMzCXAzTHc7Der9TFBcR/n3dpYVIYM23QXvcABYF3kCiTDM7w543xk2K7SqbTXem47lJ7P/lG51hflw5N5dnNBMyA0UcX6aSzw+8JJSXthpQ3CVnU9OgEnxrUzaiqxSj9W53k9r3Ff6DjQRPLlV4UOvSSKbrV3gR9Nw7dzYIYDTCwcgZyQ7O+x7kwk0jZV1Ysru/2Q9b1adKv4QdrqaTfD7ilEQgM42ApF5QwaYEmRDk2p/TH/4fDIW2YLSTjugq6E0/PRhvgLRfRsmtZwOQ2r8lvPx6oLpDTE9dQTJUWYe7KvZSkCfwY66N+II1p51l9BaUAF+ec+AUTlJB+1jAL2KEo6m0lghcz/lNoXDeVEPu0x8rr0KzSsLB+8= ceph-11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:52:26.171 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f78e402fd50 msgr2=0x7f78e4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:26.171 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f78e402fd50 0x7f78e4032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f78ec0096a0 tx=0x7f78ec006b40).stop 2021-11-20T08:52:26.172 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 msgr2=0x7f790410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:26.172 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f790410aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f78f400b1d0 tx=0x7f78f4006ee0).stop 2021-11-20T08:52:26.172 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 shutdown_connections 2021-11-20T08:52:26.173 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f78e402fd50 0x7f78e4032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:26.173 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 --2- 172.21.15.93:0/2831731396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f790406fc30 0x7f790410aa00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:26.173 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 >> 172.21.15.93:0/2831731396 conn(0x7f79040f6310 msgr2=0x7f79040f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:26.173 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 shutdown_connections 2021-11-20T08:52:26.174 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.171+0000 7f7908be0700 1 -- 172.21.15.93:0/2831731396 wait complete. 2021-11-20T08:52:26.310 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:26 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:25.144764+0000 mgr.smithi093.qdikkm (mgr.14114 2021-11-20T08:52:26.310 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:26 smithi093 conmon[33522]: ) 6 : cephadm [INF] Generating ssh key... 2021-11-20T08:52:26.310 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:26 smithi093 conmon[33522]: audit 2021-11-20T08:52:25.250634+0000 mon.smithi093 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:26.310 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:26 smithi093 conmon[33522]: audit 2021-11-20T08:52:25.254985+0000 mon.smithi093 (mon.0) 52 : audit [INF] 2021-11-20T08:52:26.311 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:26 smithi093 conmon[33522]: from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:26.479 INFO:teuthology.orchestra.run.smithi093.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-11-20T08:52:26.479 INFO:teuthology.orchestra.run.smithi093.stderr:Adding key to root@localhost authorized_keys... 2021-11-20T08:52:26.480 INFO:teuthology.orchestra.run.smithi093.stderr:Adding host smithi093... 2021-11-20T08:52:26.481 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi093 172.21.15.93 2021-11-20T08:52:26.923 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.914+0000 7f05b6727700 1 Processor -- start 2021-11-20T08:52:26.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b6727700 1 -- start start 2021-11-20T08:52:26.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b6727700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b00fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:26.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b6727700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b00fc8b0 con 0x7f05b00fbfa0 2021-11-20T08:52:26.924 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b5725700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b00fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:26.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b5725700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b00fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42198/0 (socket says 172.21.15.93:42198) 2021-11-20T08:52:26.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b5725700 1 -- 172.21.15.93:0/3716096609 learned_addr learned my addr 172.21.15.93:0/3716096609 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:26.925 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b5725700 1 -- 172.21.15.93:0/3716096609 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00fc9f0 con 0x7f05b00fbfa0 2021-11-20T08:52:26.926 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.915+0000 7f05b5725700 1 --2- 172.21.15.93:0/3716096609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b00fc3c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f05ac00b5f0 tx=0x7f05ac006ee0).ready entity=mon.0 client_cookie=595a8e00aad8f18b server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:26.927 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f059ffff700 1 -- 172.21.15.93:0/3716096609 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac00d600 con 0x7f05b00fbfa0 2021-11-20T08:52:26.927 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f059ffff700 1 -- 172.21.15.93:0/3716096609 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f05ac00dbd0 con 0x7f05b00fbfa0 2021-11-20T08:52:26.927 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f059ffff700 1 -- 172.21.15.93:0/3716096609 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac011c00 con 0x7f05b00fbfa0 2021-11-20T08:52:26.927 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f05b6727700 1 -- 172.21.15.93:0/3716096609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 msgr2=0x7f05b00fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:26.927 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f05b6727700 1 --2- 172.21.15.93:0/3716096609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b00fc3c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f05ac00b5f0 tx=0x7f05ac006ee0).stop 2021-11-20T08:52:26.928 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f05b6727700 1 -- 172.21.15.93:0/3716096609 shutdown_connections 2021-11-20T08:52:26.928 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f05b6727700 1 --2- 172.21.15.93:0/3716096609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b00fc3c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:26.928 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f05b6727700 1 -- 172.21.15.93:0/3716096609 >> 172.21.15.93:0/3716096609 conn(0x7f05b00f5870 msgr2=0x7f05b00f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:26.929 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f05b6727700 1 -- 172.21.15.93:0/3716096609 shutdown_connections 2021-11-20T08:52:26.929 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.916+0000 7f05b6727700 1 -- 172.21.15.93:0/3716096609 wait complete. 2021-11-20T08:52:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b6727700 1 Processor -- start 2021-11-20T08:52:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b6727700 1 -- start start 2021-11-20T08:52:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b6727700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b010ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b6727700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b010cef0 con 0x7f05b00fbfa0 2021-11-20T08:52:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b5725700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b010ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b5725700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b010ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42200/0 (socket says 172.21.15.93:42200) 2021-11-20T08:52:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b5725700 1 -- 172.21.15.93:0/3138827078 learned_addr learned my addr 172.21.15.93:0/3138827078 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b5725700 1 -- 172.21.15.93:0/3138827078 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac00b200 con 0x7f05b00fbfa0 2021-11-20T08:52:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.917+0000 7f05b5725700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b010ca00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f05ac00e040 tx=0x7f05ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.918+0000 7f059dffb700 1 -- 172.21.15.93:0/3138827078 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac01b410 con 0x7f05b00fbfa0 2021-11-20T08:52:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.918+0000 7f059dffb700 1 -- 172.21.15.93:0/3138827078 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f05ac01b9e0 con 0x7f05b00fbfa0 2021-11-20T08:52:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.918+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b010d0f0 con 0x7f05b00fbfa0 2021-11-20T08:52:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.918+0000 7f059dffb700 1 -- 172.21.15.93:0/3138827078 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac01a770 con 0x7f05b00fbfa0 2021-11-20T08:52:26.933 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.918+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b010d500 con 0x7f05b00fbfa0 2021-11-20T08:52:26.933 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.919+0000 7f059dffb700 1 -- 172.21.15.93:0/3138827078 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f05ac00f070 con 0x7f05b00fbfa0 2021-11-20T08:52:26.933 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.919+0000 7f059dffb700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f059802fd50 0x7f0598032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:26.933 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.919+0000 7f059dffb700 1 -- 172.21.15.93:0/3138827078 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f05ac041030 con 0x7f05b00fbfa0 2021-11-20T08:52:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.919+0000 7f05b4f24700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f059802fd50 0x7f0598032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.919+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b0059100 con 0x7f05b00fbfa0 2021-11-20T08:52:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.919+0000 7f05b4f24700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f059802fd50 0x7f0598032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f05a00096a0 tx=0x7f05a0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:26.923+0000 7f059dffb700 1 -- 172.21.15.93:0/3138827078 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05ac016070 con 0x7f05b00fbfa0 2021-11-20T08:52:27.094 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.094+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi093", "addr": "172.21.15.93", "target": ["mon-mgr", ""]}) v1 -- 0x7f05b0106840 con 0x7f059802fd50 2021-11-20T08:52:27.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:27 smithi093 conmon[33522]: audit 2021-11-20T08:52:26.170322+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:52:27.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:27 smithi093 conmon[33522]: mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:27.767 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Added host 'smithi093' 2021-11-20T08:52:27.768 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.763+0000 7f059dffb700 1 -- 172.21.15.93:0/3138827078 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f05b0106840 con 0x7f059802fd50 2021-11-20T08:52:27.768 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.765+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f059802fd50 msgr2=0x7f0598032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:27.769 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.765+0000 7f05b6727700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f059802fd50 0x7f0598032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f05a00096a0 tx=0x7f05a0006b40).stop 2021-11-20T08:52:27.769 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.765+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 msgr2=0x7f05b010ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:27.770 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.766+0000 7f05b6727700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b010ca00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f05ac00e040 tx=0x7f05ac006ee0).stop 2021-11-20T08:52:27.770 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.766+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 shutdown_connections 2021-11-20T08:52:27.770 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.766+0000 7f05b6727700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f059802fd50 0x7f0598032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:27.771 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.766+0000 7f05b6727700 1 --2- 172.21.15.93:0/3138827078 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05b00fbfa0 0x7f05b010ca00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:27.771 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.766+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 >> 172.21.15.93:0/3138827078 conn(0x7f05b00f5870 msgr2=0x7f05b00f73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:27.772 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.766+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 shutdown_connections 2021-11-20T08:52:27.772 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:27.766+0000 7f05b6727700 1 -- 172.21.15.93:0/3138827078 wait complete. 2021-11-20T08:52:28.120 INFO:teuthology.orchestra.run.smithi093.stderr:Deploying mon service with default placement... 2021-11-20T08:52:28.121 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2021-11-20T08:52:28.517 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.507+0000 7f484f527700 1 Processor -- start 2021-11-20T08:52:28.520 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.507+0000 7f484f527700 1 -- start start 2021-11-20T08:52:28.520 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.507+0000 7f484f527700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f48480fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:28.522 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.507+0000 7f484f527700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48480fc2b0 con 0x7f48480fb9a0 2021-11-20T08:52:28.522 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.507+0000 7f484e525700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f48480fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:28.523 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.507+0000 7f484e525700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f48480fbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42210/0 (socket says 172.21.15.93:42210) 2021-11-20T08:52:28.523 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.507+0000 7f484e525700 1 -- 172.21.15.93:0/412035219 learned_addr learned my addr 172.21.15.93:0/412035219 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:28.524 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.508+0000 7f484e525700 1 -- 172.21.15.93:0/412035219 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480fc3f0 con 0x7f48480fb9a0 2021-11-20T08:52:28.524 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.508+0000 7f484e525700 1 --2- 172.21.15.93:0/412035219 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f48480fbdc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f484400b5f0 tx=0x7f4844006ee0).ready entity=mon.0 client_cookie=ac5884e49d9a1586 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:28.524 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.508+0000 7f484d523700 1 -- 172.21.15.93:0/412035219 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f484400d600 con 0x7f48480fb9a0 2021-11-20T08:52:28.525 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.508+0000 7f484d523700 1 -- 172.21.15.93:0/412035219 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f484400dbd0 con 0x7f48480fb9a0 2021-11-20T08:52:28.525 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.508+0000 7f484d523700 1 -- 172.21.15.93:0/412035219 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4844011c00 con 0x7f48480fb9a0 2021-11-20T08:52:28.525 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 -- 172.21.15.93:0/412035219 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 msgr2=0x7f48480fbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:28.526 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 --2- 172.21.15.93:0/412035219 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f48480fbdc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f484400b5f0 tx=0x7f4844006ee0).stop 2021-11-20T08:52:28.526 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 -- 172.21.15.93:0/412035219 shutdown_connections 2021-11-20T08:52:28.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 --2- 172.21.15.93:0/412035219 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f48480fbdc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:28.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 -- 172.21.15.93:0/412035219 >> 172.21.15.93:0/412035219 conn(0x7f48480f71e0 msgr2=0x7f48480f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:28.527 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 -- 172.21.15.93:0/412035219 shutdown_connections 2021-11-20T08:52:28.528 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 -- 172.21.15.93:0/412035219 wait complete. 2021-11-20T08:52:28.528 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 Processor -- start 2021-11-20T08:52:28.528 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.509+0000 7f484f527700 1 -- start start 2021-11-20T08:52:28.529 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484f527700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f484810a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:28.529 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484f527700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484810aca0 con 0x7f48480fb9a0 2021-11-20T08:52:28.529 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484e525700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f484810a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:28.530 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484e525700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f484810a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42212/0 (socket says 172.21.15.93:42212) 2021-11-20T08:52:28.530 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484e525700 1 -- 172.21.15.93:0/1036608672 learned_addr learned my addr 172.21.15.93:0/1036608672 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:28.531 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484e525700 1 -- 172.21.15.93:0/1036608672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484400b200 con 0x7f48480fb9a0 2021-11-20T08:52:28.531 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484e525700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f484810a7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f484400e040 tx=0x7f4844006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:28.531 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f4836ffd700 1 -- 172.21.15.93:0/1036608672 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f484401b410 con 0x7f48480fb9a0 2021-11-20T08:52:28.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f4836ffd700 1 -- 172.21.15.93:0/1036608672 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f484401b9e0 con 0x7f48480fb9a0 2021-11-20T08:52:28.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f4836ffd700 1 -- 172.21.15.93:0/1036608672 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f484401a820 con 0x7f48480fb9a0 2021-11-20T08:52:28.532 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484810aea0 con 0x7f48480fb9a0 2021-11-20T08:52:28.533 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.510+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484810b2b0 con 0x7f48480fb9a0 2021-11-20T08:52:28.533 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.511+0000 7f4836ffd700 1 -- 172.21.15.93:0/1036608672 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f484400f070 con 0x7f48480fb9a0 2021-11-20T08:52:28.533 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.511+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4848045ad0 con 0x7f48480fb9a0 2021-11-20T08:52:28.534 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.511+0000 7f4836ffd700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f483002fdb0 0x7f4830032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:28.534 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.511+0000 7f4836ffd700 1 -- 172.21.15.93:0/1036608672 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f48440413d0 con 0x7f48480fb9a0 2021-11-20T08:52:28.534 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.512+0000 7f484dd24700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f483002fdb0 0x7f4830032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:28.534 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.512+0000 7f484dd24700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f483002fdb0 0x7f4830032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f48380096a0 tx=0x7f4838006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:28.535 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.515+0000 7f4836ffd700 1 -- 172.21.15.93:0/1036608672 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4844022310 con 0x7f48480fb9a0 2021-11-20T08:52:28.701 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.700+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f484810bb90 con 0x7f483002fdb0 2021-11-20T08:52:28.704 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.705+0000 7f4836ffd700 1 -- 172.21.15.93:0/1036608672 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f484810bb90 con 0x7f483002fdb0 2021-11-20T08:52:28.705 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Scheduled mon update... 2021-11-20T08:52:28.708 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f483002fdb0 msgr2=0x7f4830032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:28.708 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f483002fdb0 0x7f4830032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f48380096a0 tx=0x7f4838006b40).stop 2021-11-20T08:52:28.708 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 msgr2=0x7f484810a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:28.708 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f484810a7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f484400e040 tx=0x7f4844006ee0).stop 2021-11-20T08:52:28.711 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 shutdown_connections 2021-11-20T08:52:28.711 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f483002fdb0 0x7f4830032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:28.711 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 --2- 172.21.15.93:0/1036608672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48480fb9a0 0x7f484810a7b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:28.713 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 >> 172.21.15.93:0/1036608672 conn(0x7f48480f71e0 msgr2=0x7f48480f8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:28.713 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 shutdown_connections 2021-11-20T08:52:28.714 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:28.706+0000 7f484f527700 1 -- 172.21.15.93:0/1036608672 wait complete. 2021-11-20T08:52:29.100 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:28 smithi093 conmon[33522]: audit 2021-11-20T08:52:27.096646+0000 mgr.smithi093.qdikkm (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi093", "addr": "172.21.15.93", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:29.101 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:28 smithi093 conmon[33522]: audit 2021-11-20T08:52:27.764470+0000 mon.smithi093 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:29.101 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:28 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:27.764790+0000 mgr.smithi093.qdikkm (mgr.14114) 9 : cephadm [INF] Added host smithi093 2021-11-20T08:52:29.101 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:28 smithi093 conmon[33522]: audit 2021-11-20T08:52:27.765025+0000 mon.smithi093 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:29.102 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:28 smithi093 conmon[33522]: audit 2021-11-20T08:52:28.706480+0000 mon.smithi093 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:29.196 INFO:teuthology.orchestra.run.smithi093.stderr:Deploying mgr service with default placement... 2021-11-20T08:52:29.196 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2021-11-20T08:52:29.750 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.745+0000 7fef5dd6e700 1 Processor -- start 2021-11-20T08:52:29.750 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.745+0000 7fef5dd6e700 1 -- start start 2021-11-20T08:52:29.750 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.745+0000 7fef5dd6e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef580fe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:29.751 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.745+0000 7fef5dd6e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580feb80 con 0x7fef5806fbf0 2021-11-20T08:52:29.751 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.745+0000 7fef5cd6c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef580fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:29.751 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.746+0000 7fef5cd6c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef580fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42220/0 (socket says 172.21.15.93:42220) 2021-11-20T08:52:29.751 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.746+0000 7fef5cd6c700 1 -- 172.21.15.93:0/867264083 learned_addr learned my addr 172.21.15.93:0/867264083 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:29.752 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.746+0000 7fef5cd6c700 1 -- 172.21.15.93:0/867264083 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580fecc0 con 0x7fef5806fbf0 2021-11-20T08:52:29.752 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.746+0000 7fef5cd6c700 1 --2- 172.21.15.93:0/867264083 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef580fe690 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fef5000b5f0 tx=0x7fef50006ee0).ready entity=mon.0 client_cookie=f8849a793164a4ad server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:29.752 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.746+0000 7fef577fe700 1 -- 172.21.15.93:0/867264083 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef5000d600 con 0x7fef5806fbf0 2021-11-20T08:52:29.752 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.746+0000 7fef577fe700 1 -- 172.21.15.93:0/867264083 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fef5000dbd0 con 0x7fef5806fbf0 2021-11-20T08:52:29.753 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.746+0000 7fef577fe700 1 -- 172.21.15.93:0/867264083 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef50011c30 con 0x7fef5806fbf0 2021-11-20T08:52:29.753 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 -- 172.21.15.93:0/867264083 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 msgr2=0x7fef580fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:29.753 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 --2- 172.21.15.93:0/867264083 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef580fe690 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fef5000b5f0 tx=0x7fef50006ee0).stop 2021-11-20T08:52:29.753 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 -- 172.21.15.93:0/867264083 shutdown_connections 2021-11-20T08:52:29.753 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 --2- 172.21.15.93:0/867264083 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef580fe690 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:29.754 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 -- 172.21.15.93:0/867264083 >> 172.21.15.93:0/867264083 conn(0x7fef580f6130 msgr2=0x7fef580f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:29.754 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 -- 172.21.15.93:0/867264083 shutdown_connections 2021-11-20T08:52:29.754 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 -- 172.21.15.93:0/867264083 wait complete. 2021-11-20T08:52:29.754 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 Processor -- start 2021-11-20T08:52:29.755 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.747+0000 7fef5dd6e700 1 -- start start 2021-11-20T08:52:29.755 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5dd6e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef5810a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:29.755 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5dd6e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5810ac60 con 0x7fef5806fbf0 2021-11-20T08:52:29.755 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5cd6c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef5810a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:29.755 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5cd6c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef5810a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42222/0 (socket says 172.21.15.93:42222) 2021-11-20T08:52:29.756 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5cd6c700 1 -- 172.21.15.93:0/3028177018 learned_addr learned my addr 172.21.15.93:0/3028177018 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:29.756 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5cd6c700 1 -- 172.21.15.93:0/3028177018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5000b200 con 0x7fef5806fbf0 2021-11-20T08:52:29.756 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5cd6c700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef5810a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fef50011850 tx=0x7fef50006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:29.756 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef557fa700 1 -- 172.21.15.93:0/3028177018 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef5001b570 con 0x7fef5806fbf0 2021-11-20T08:52:29.757 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5810ae60 con 0x7fef5806fbf0 2021-11-20T08:52:29.757 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef557fa700 1 -- 172.21.15.93:0/3028177018 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fef5001bb40 con 0x7fef5806fbf0 2021-11-20T08:52:29.757 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef557fa700 1 -- 172.21.15.93:0/3028177018 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef5001aa10 con 0x7fef5806fbf0 2021-11-20T08:52:29.757 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.748+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5810b270 con 0x7fef5806fbf0 2021-11-20T08:52:29.757 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.749+0000 7fef557fa700 1 -- 172.21.15.93:0/3028177018 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fef5000f070 con 0x7fef5806fbf0 2021-11-20T08:52:29.758 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.749+0000 7fef557fa700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fef38034160 0x7fef38036620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:29.758 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.750+0000 7fef557fa700 1 -- 172.21.15.93:0/3028177018 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fef50041120 con 0x7fef5806fbf0 2021-11-20T08:52:29.758 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.749+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef58045ad0 con 0x7fef5806fbf0 2021-11-20T08:52:29.758 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.750+0000 7fef57fff700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fef38034160 0x7fef38036620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:29.759 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.750+0000 7fef57fff700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fef38034160 0x7fef38036620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fef480089e0 tx=0x7fef48006ec0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:29.759 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.754+0000 7fef557fa700 1 -- 172.21.15.93:0/3028177018 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef50016070 con 0x7fef5806fbf0 2021-11-20T08:52:29.939 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:29 smithi093 conmon[33522]: audit 2021-11-20T08:52:28.703504+0000 2021-11-20T08:52:29.939 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:29 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:29.940 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:29 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:28.704261+0000 mgr.smithi093.qdikkm (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2021-11-20T08:52:29.940 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.939+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fef5810bb50 con 0x7fef38034160 2021-11-20T08:52:29.943 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.943+0000 7fef557fa700 1 -- 172.21.15.93:0/3028177018 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fef5810bb50 con 0x7fef38034160 2021-11-20T08:52:29.943 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-11-20T08:52:29.945 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fef38034160 msgr2=0x7fef38036620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:29.946 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fef38034160 0x7fef38036620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fef480089e0 tx=0x7fef48006ec0).stop 2021-11-20T08:52:29.946 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 msgr2=0x7fef5810a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:29.946 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef5810a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fef50011850 tx=0x7fef50006ee0).stop 2021-11-20T08:52:29.946 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 shutdown_connections 2021-11-20T08:52:29.946 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fef38034160 0x7fef38036620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:29.947 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 --2- 172.21.15.93:0/3028177018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fef5806fbf0 0x7fef5810a770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:29.947 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 >> 172.21.15.93:0/3028177018 conn(0x7fef580f6130 msgr2=0x7fef580f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:29.947 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 shutdown_connections 2021-11-20T08:52:29.947 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:29.945+0000 7fef5dd6e700 1 -- 172.21.15.93:0/3028177018 wait complete. 2021-11-20T08:52:30.384 INFO:teuthology.orchestra.run.smithi093.stderr:Deploying crash service with default placement... 2021-11-20T08:52:30.384 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2021-11-20T08:52:30.828 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.825+0000 7fb13be79700 1 Processor -- start 2021-11-20T08:52:30.829 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.825+0000 7fb13be79700 1 -- start start 2021-11-20T08:52:30.829 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.825+0000 7fb13be79700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb1340fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:30.829 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.825+0000 7fb13be79700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1340fcb20 con 0x7fb1340fc210 2021-11-20T08:52:30.830 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb139c15700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb1340fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:30.830 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb139c15700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb1340fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42226/0 (socket says 172.21.15.93:42226) 2021-11-20T08:52:30.830 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb139c15700 1 -- 172.21.15.93:0/3121190328 learned_addr learned my addr 172.21.15.93:0/3121190328 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:30.831 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb139c15700 1 -- 172.21.15.93:0/3121190328 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340fcc60 con 0x7fb1340fc210 2021-11-20T08:52:30.831 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb139c15700 1 --2- 172.21.15.93:0/3121190328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb1340fc630 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb12c00b5f0 tx=0x7fb12c006ee0).ready entity=mon.0 client_cookie=a7f3d634d31b8224 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:30.831 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb138c13700 1 -- 172.21.15.93:0/3121190328 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c00d600 con 0x7fb1340fc210 2021-11-20T08:52:30.832 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb138c13700 1 -- 172.21.15.93:0/3121190328 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb12c00dc10 con 0x7fb1340fc210 2021-11-20T08:52:30.832 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.826+0000 7fb138c13700 1 -- 172.21.15.93:0/3121190328 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c011cd0 con 0x7fb1340fc210 2021-11-20T08:52:30.832 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 -- 172.21.15.93:0/3121190328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 msgr2=0x7fb1340fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:30.832 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 --2- 172.21.15.93:0/3121190328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb1340fc630 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb12c00b5f0 tx=0x7fb12c006ee0).stop 2021-11-20T08:52:30.833 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 -- 172.21.15.93:0/3121190328 shutdown_connections 2021-11-20T08:52:30.833 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 --2- 172.21.15.93:0/3121190328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb1340fc630 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:30.833 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 -- 172.21.15.93:0/3121190328 >> 172.21.15.93:0/3121190328 conn(0x7fb1340f5a80 msgr2=0x7fb1340f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:30.833 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 -- 172.21.15.93:0/3121190328 shutdown_connections 2021-11-20T08:52:30.833 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 -- 172.21.15.93:0/3121190328 wait complete. 2021-11-20T08:52:30.834 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 Processor -- start 2021-11-20T08:52:30.834 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.827+0000 7fb13be79700 1 -- start start 2021-11-20T08:52:30.834 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb13be79700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb13410cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:30.834 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb13be79700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13410d1d0 con 0x7fb1340fc210 2021-11-20T08:52:30.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb139c15700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb13410cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:30.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb139c15700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb13410cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42228/0 (socket says 172.21.15.93:42228) 2021-11-20T08:52:30.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb139c15700 1 -- 172.21.15.93:0/2477434796 learned_addr learned my addr 172.21.15.93:0/2477434796 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:30.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb139c15700 1 -- 172.21.15.93:0/2477434796 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c00b200 con 0x7fb1340fc210 2021-11-20T08:52:30.835 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb139c15700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb13410cce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb12c00dfb0 tx=0x7fb12c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:30.836 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb1327fc700 1 -- 172.21.15.93:0/2477434796 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c011df0 con 0x7fb1340fc210 2021-11-20T08:52:30.836 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13410d3d0 con 0x7fb1340fc210 2021-11-20T08:52:30.836 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb1327fc700 1 -- 172.21.15.93:0/2477434796 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb12c01b8c0 con 0x7fb1340fc210 2021-11-20T08:52:30.836 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13410d7e0 con 0x7fb1340fc210 2021-11-20T08:52:30.837 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.828+0000 7fb1327fc700 1 -- 172.21.15.93:0/2477434796 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12c01a7c0 con 0x7fb1340fc210 2021-11-20T08:52:30.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.829+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb134045ad0 con 0x7fb1340fc210 2021-11-20T08:52:30.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.830+0000 7fb1327fc700 1 -- 172.21.15.93:0/2477434796 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb12c00f070 con 0x7fb1340fc210 2021-11-20T08:52:30.838 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.830+0000 7fb1327fc700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12002fd50 0x7fb120032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:30.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.830+0000 7fb1327fc700 1 -- 172.21.15.93:0/2477434796 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb12c0415a0 con 0x7fb1340fc210 2021-11-20T08:52:30.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.830+0000 7fb139414700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12002fd50 0x7fb120032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:30.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.830+0000 7fb139414700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12002fd50 0x7fb120032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb1280096a0 tx=0x7fb128006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:30.839 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:30.834+0000 7fb1327fc700 1 -- 172.21.15.93:0/2477434796 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb12c022d20 con 0x7fb1340fc210 2021-11-20T08:52:31.034 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.034+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb13410e040 con 0x7fb12002fd50 2021-11-20T08:52:31.039 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.039+0000 7fb1327fc700 1 -- 172.21.15.93:0/2477434796 <== mgr.14114 v2:172.21.15.93:6800/3748247363 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb13410e040 con 0x7fb12002fd50 2021-11-20T08:52:31.040 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Scheduled crash update... 2021-11-20T08:52:31.042 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12002fd50 msgr2=0x7fb120032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:31.042 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12002fd50 0x7fb120032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb1280096a0 tx=0x7fb128006b40).stop 2021-11-20T08:52:31.043 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 msgr2=0x7fb13410cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:31.043 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb13410cce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb12c00dfb0 tx=0x7fb12c006ee0).stop 2021-11-20T08:52:31.043 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 shutdown_connections 2021-11-20T08:52:31.043 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7fb12002fd50 0x7fb120032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:31.044 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 --2- 172.21.15.93:0/2477434796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1340fc210 0x7fb13410cce0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:31.044 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 >> 172.21.15.93:0/2477434796 conn(0x7fb1340f5a80 msgr2=0x7fb1340f7630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:31.044 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.041+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 shutdown_connections 2021-11-20T08:52:31.045 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.042+0000 7fb13be79700 1 -- 172.21.15.93:0/2477434796 wait complete. 2021-11-20T08:52:31.184 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: audit 2021-11-20T08:52:29.941875+0000 mgr.smithi093.qdikkm (mgr.14114 2021-11-20T08:52:31.185 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: ) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:31.185 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:29.942530+0000 mgr.smithi093.qdikkm (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2021-11-20T08:52:31.186 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: audit 2021-11-20T08:52:29.944917+0000 mon.smithi093 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:31.186 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:52:31.187 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: 08:52:30.147657+0000 mon.smithi093 (mon.0) 57 : audit [INF] 2021-11-20T08:52:31.187 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:31.187 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:52:31.188 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: 11-20T08:52:30.541513+0000 mon.smithi093 (mon.0) 58 : audit 2021-11-20T08:52:31.188 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:30 smithi093 conmon[33522]: [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:31.319 INFO:teuthology.orchestra.run.smithi093.stderr:Enabling mgr prometheus module... 2021-11-20T08:52:31.319 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2021-11-20T08:52:31.848 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.844+0000 7ff4a3391700 1 Processor -- start 2021-11-20T08:52:31.848 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.844+0000 7ff4a3391700 1 -- start start 2021-11-20T08:52:31.848 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.844+0000 7ff4a3391700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c100ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:31.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.844+0000 7ff4a3391700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c0fc620 con 0x7ff49c06fcd0 2021-11-20T08:52:31.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff4a112d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c100ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:31.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff4a112d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c100ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42232/0 (socket says 172.21.15.93:42232) 2021-11-20T08:52:31.849 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff4a112d700 1 -- 172.21.15.93:0/365675 learned_addr learned my addr 172.21.15.93:0/365675 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:31.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff4a112d700 1 -- 172.21.15.93:0/365675 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c101090 con 0x7ff49c06fcd0 2021-11-20T08:52:31.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff4a112d700 1 --2- 172.21.15.93:0/365675 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c100ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff49800b5f0 tx=0x7ff498006ee0).ready entity=mon.0 client_cookie=5574d903fed52569 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:31.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff48bfff700 1 -- 172.21.15.93:0/365675 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff49800d600 con 0x7ff49c06fcd0 2021-11-20T08:52:31.850 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff48bfff700 1 -- 172.21.15.93:0/365675 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff49800dc10 con 0x7ff49c06fcd0 2021-11-20T08:52:31.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.845+0000 7ff48bfff700 1 -- 172.21.15.93:0/365675 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff498011cd0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.846+0000 7ff4a3391700 1 -- 172.21.15.93:0/365675 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 msgr2=0x7ff49c100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:31.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.846+0000 7ff4a3391700 1 --2- 172.21.15.93:0/365675 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c100ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff49800b5f0 tx=0x7ff498006ee0).stop 2021-11-20T08:52:31.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.846+0000 7ff4a3391700 1 -- 172.21.15.93:0/365675 shutdown_connections 2021-11-20T08:52:31.851 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.846+0000 7ff4a3391700 1 --2- 172.21.15.93:0/365675 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c100ba0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:31.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.846+0000 7ff4a3391700 1 -- 172.21.15.93:0/365675 >> 172.21.15.93:0/365675 conn(0x7ff49c0f6280 msgr2=0x7ff49c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:31.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.846+0000 7ff4a3391700 1 -- 172.21.15.93:0/365675 shutdown_connections 2021-11-20T08:52:31.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.846+0000 7ff4a3391700 1 -- 172.21.15.93:0/365675 wait complete. 2021-11-20T08:52:31.852 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a3391700 1 Processor -- start 2021-11-20T08:52:31.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a3391700 1 -- start start 2021-11-20T08:52:31.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a3391700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c10ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:31.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a3391700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c10d1c0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a112d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c10ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:31.853 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a112d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c10ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42234/0 (socket says 172.21.15.93:42234) 2021-11-20T08:52:31.854 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a112d700 1 -- 172.21.15.93:0/2524894826 learned_addr learned my addr 172.21.15.93:0/2524894826 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:31.854 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a112d700 1 -- 172.21.15.93:0/2524894826 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49800b200 con 0x7ff49c06fcd0 2021-11-20T08:52:31.854 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff4a112d700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c10ccd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff49800bf40 tx=0x7ff498006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:31.854 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.847+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff498011df0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.848+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff49801b8c0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.848+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c10d3c0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.848+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff49801a8e0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.855 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.848+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c10d7d0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.856 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.849+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff49800f070 con 0x7ff49c06fcd0 2021-11-20T08:52:31.857 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.849+0000 7ff489ffb700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7ff48402fdb0 0x7ff484032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:31.857 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.849+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff498040db0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.857 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.849+0000 7ff4a092c700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7ff48402fdb0 0x7ff484032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:31.857 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.849+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c045ad0 con 0x7ff49c06fcd0 2021-11-20T08:52:31.858 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.849+0000 7ff4a092c700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7ff48402fdb0 0x7ff484032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff48c0096a0 tx=0x7ff48c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:31.858 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:31.853+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff498016070 con 0x7ff49c06fcd0 2021-11-20T08:52:32.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:32.053+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7ff49c106a90 con 0x7ff49c06fcd0 2021-11-20T08:52:32.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:32 smithi093 conmon[33522]: audit 2021-11-20T08:52:31.037129+0000 mgr.smithi093.qdikkm (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:32.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:32 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:31.038014+0000 mgr.smithi093.qdikkm (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2021-11-20T08:52:32.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:32 smithi093 conmon[33522]: audit 2021-11-20T08:52:31.041125+0000 mon.smithi093 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:33.043 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.042+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7ff49801aa40 con 0x7ff49c06fcd0 2021-11-20T08:52:33.043 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.043+0000 7ff489ffb700 1 -- 172.21.15.93:0/2524894826 <== mon.0 v2:172.21.15.93:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7ff498022530 con 0x7ff49c06fcd0 2021-11-20T08:52:33.047 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7ff48402fdb0 msgr2=0x7ff484032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:33.047 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7ff48402fdb0 0x7ff484032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff48c0096a0 tx=0x7ff48c006b40).stop 2021-11-20T08:52:33.047 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 msgr2=0x7ff49c10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:33.048 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c10ccd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff49800bf40 tx=0x7ff498006ee0).stop 2021-11-20T08:52:33.048 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 shutdown_connections 2021-11-20T08:52:33.048 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7ff48402fdb0 0x7ff484032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:33.048 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 --2- 172.21.15.93:0/2524894826 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff49c06fcd0 0x7ff49c10ccd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:33.048 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 >> 172.21.15.93:0/2524894826 conn(0x7ff49c0f6280 msgr2=0x7ff49c0f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:33.049 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 shutdown_connections 2021-11-20T08:52:33.049 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.046+0000 7ff4a3391700 1 -- 172.21.15.93:0/2524894826 wait complete. 2021-11-20T08:52:33.185 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:33 smithi093 conmon[33522]: audit 2021-11-20T08:52:32.055789+0000 mon.smithi093 ( 2021-11-20T08:52:33.185 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:33 smithi093 conmon[33522]: mon.0) 60 : audit [INF] from='client.? 172.21.15.93:0/2524894826' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-11-20T08:52:33.185 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:33 smithi093 conmon[33522]: audit 2021-11-20T08:52:32.133465+0000 mon.smithi093 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.93:0/1048608711' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:33.354 INFO:teuthology.orchestra.run.smithi093.stderr:Deploying prometheus service with default placement... 2021-11-20T08:52:33.355 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2021-11-20T08:52:33.816 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.810+0000 7f33da01c700 1 Processor -- start 2021-11-20T08:52:33.816 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.810+0000 7f33da01c700 1 -- start start 2021-11-20T08:52:33.816 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.811+0000 7f33da01c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d40fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:33.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.811+0000 7f33da01c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40fed20 con 0x7f33d406fcd0 2021-11-20T08:52:33.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.811+0000 7f33d37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d40fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:33.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.811+0000 7f33d37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d40fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42248/0 (socket says 172.21.15.93:42248) 2021-11-20T08:52:33.817 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.811+0000 7f33d37fe700 1 -- 172.21.15.93:0/3782132982 learned_addr learned my addr 172.21.15.93:0/3782132982 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:33.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.811+0000 7f33d37fe700 1 -- 172.21.15.93:0/3782132982 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d40fee60 con 0x7f33d406fcd0 2021-11-20T08:52:33.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.811+0000 7f33d37fe700 1 --2- 172.21.15.93:0/3782132982 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d40fe7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f33c800b5f0 tx=0x7f33c8006ee0).ready entity=mon.0 client_cookie=11767e0daec1abf7 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:33.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33d27fc700 1 -- 172.21.15.93:0/3782132982 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c800d600 con 0x7f33d406fcd0 2021-11-20T08:52:33.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33d27fc700 1 -- 172.21.15.93:0/3782132982 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c800dc10 con 0x7f33d406fcd0 2021-11-20T08:52:33.818 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33d27fc700 1 -- 172.21.15.93:0/3782132982 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c8011cd0 con 0x7f33d406fcd0 2021-11-20T08:52:33.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33da01c700 1 -- 172.21.15.93:0/3782132982 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 msgr2=0x7f33d40fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:33.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33da01c700 1 --2- 172.21.15.93:0/3782132982 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d40fe7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f33c800b5f0 tx=0x7f33c8006ee0).stop 2021-11-20T08:52:33.819 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33da01c700 1 -- 172.21.15.93:0/3782132982 shutdown_connections 2021-11-20T08:52:33.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33da01c700 1 --2- 172.21.15.93:0/3782132982 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d40fe7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:33.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.812+0000 7f33da01c700 1 -- 172.21.15.93:0/3782132982 >> 172.21.15.93:0/3782132982 conn(0x7f33d40f6280 msgr2=0x7f33d40f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:33.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.813+0000 7f33da01c700 1 -- 172.21.15.93:0/3782132982 shutdown_connections 2021-11-20T08:52:33.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.813+0000 7f33da01c700 1 -- 172.21.15.93:0/3782132982 wait complete. 2021-11-20T08:52:33.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.813+0000 7f33da01c700 1 Processor -- start 2021-11-20T08:52:33.821 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.813+0000 7f33da01c700 1 -- start start 2021-11-20T08:52:33.821 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.813+0000 7f33da01c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:33.821 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.813+0000 7f33da01c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d410af30 con 0x7f33d406fcd0 2021-11-20T08:52:33.821 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33d37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:33.822 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33d37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42250/0 (socket says 172.21.15.93:42250) 2021-11-20T08:52:33.822 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33d37fe700 1 -- 172.21.15.93:0/2966336220 learned_addr learned my addr 172.21.15.93:0/2966336220 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:33.822 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33d37fe700 1 -- 172.21.15.93:0/2966336220 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c800b200 con 0x7f33d406fcd0 2021-11-20T08:52:33.823 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33d37fe700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d410aa40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f33c800dfb0 tx=0x7f33c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:33.823 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c8011df0 con 0x7f33d406fcd0 2021-11-20T08:52:33.823 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c801b8c0 con 0x7f33d406fcd0 2021-11-20T08:52:33.824 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c801a800 con 0x7f33d406fcd0 2021-11-20T08:52:33.824 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d410b130 con 0x7f33d406fcd0 2021-11-20T08:52:33.824 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.814+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d410b540 con 0x7f33d406fcd0 2021-11-20T08:52:33.825 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.815+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f33c800f070 con 0x7f33d406fcd0 2021-11-20T08:52:33.825 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.815+0000 7f33bbfff700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:33.826 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.815+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f33c8040e30 con 0x7f33d406fcd0 2021-11-20T08:52:33.826 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.815+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d4045ad0 con 0x7f33d406fcd0 2021-11-20T08:52:33.826 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.815+0000 7f33d2ffd700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 msgr2=0x7f33c00322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3748247363 2021-11-20T08:52:33.826 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.815+0000 7f33d2ffd700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T08:52:33.827 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:33.820+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33c8016070 con 0x7f33d406fcd0 2021-11-20T08:52:34.003 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:34.002+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 --> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f33d410be20 con 0x7f33c002fe00 2021-11-20T08:52:34.016 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:34.016+0000 7f33d2ffd700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 msgr2=0x7f33c00322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3748247363 2021-11-20T08:52:34.016 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:34.016+0000 7f33d2ffd700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T08:52:34.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:34 smithi093 conmon[33522]: audit 2021-11-20T08:52:33.045678+0000 mon.smithi093 (mon.0) 62 : audit 2021-11-20T08:52:34.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:34 smithi093 conmon[33522]: [INF] from='client.? 172.21.15.93:0/2524894826' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-11-20T08:52:34.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:34 smithi093 conmon[33522]: cluster 2021-11-20T08:52:33.045735+0000 mon.smithi093 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi093.qdikkm(active, since 17s) 2021-11-20T08:52:34.417 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:34.417+0000 7f33d2ffd700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 msgr2=0x7f33c00322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3748247363 2021-11-20T08:52:34.417 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:34.417+0000 7f33d2ffd700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T08:52:35.218 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:35.218+0000 7f33d2ffd700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 msgr2=0x7f33c00322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3748247363 2021-11-20T08:52:35.218 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:35.218+0000 7f33d2ffd700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T08:52:36.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:36.820+0000 7f33d2ffd700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 msgr2=0x7f33c00322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3748247363 2021-11-20T08:52:36.820 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:36.820+0000 7f33d2ffd700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T08:52:38.037 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:38.038+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f33c8019d30 con 0x7f33d406fcd0 2021-11-20T08:52:38.037 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:38.038+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 msgr2=0x7f33c00322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T08:52:38.038 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:38.038+0000 7f33bbfff700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:38.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:38 smithi093 conmon[33522]: cluster 2021-11-20T08:52:37.986790+0000 mon.smithi093 ( 2021-11-20T08:52:38.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:38 smithi093 conmon[33522]: mon.0) 64 : cluster [INF] Active manager daemon smithi093.qdikkm restarted 2021-11-20T08:52:38.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:38 smithi093 conmon[33522]: cluster 2021-11-20T08:52:37.987165+0000 mon.smithi093 (mon.0) 65 : 2021-11-20T08:52:38.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:38 smithi093 conmon[33522]: cluster [INF] Activating manager daemon smithi093.qdikkm 2021-11-20T08:52:38.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:38 smithi093 conmon[33522]: cluster 2021-11-20T08:52:37.988789+0000 2021-11-20T08:52:38.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:38 smithi093 conmon[33522]: mon.smithi093 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-11-20T08:52:39.045 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.044+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mon.0 v2:172.21.15.93:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f33c801a960 con 0x7f33d406fcd0 2021-11-20T08:52:39.045 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.044+0000 7f33bbfff700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f33c0033c20 0x7f33c0036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:39.046 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.044+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 --> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f33d410be20 con 0x7f33c0033c20 2021-11-20T08:52:39.046 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.045+0000 7f33d2ffd700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f33c0033c20 0x7f33c0036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:39.046 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.045+0000 7f33d2ffd700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f33c0033c20 0x7f33c0036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f33c40036d0 tx=0x7f33c4006ce0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:39.048 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.049+0000 7f33bbfff700 1 -- 172.21.15.93:0/2966336220 <== mgr.14142 v2:172.21.15.93:6800/3294416137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f33d410be20 con 0x7f33c0033c20 2021-11-20T08:52:39.048 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Scheduled prometheus update... 2021-11-20T08:52:39.051 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f33c0033c20 msgr2=0x7f33c0036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:39.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f33c0033c20 0x7f33c0036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f33c40036d0 tx=0x7f33c4006ce0).stop 2021-11-20T08:52:39.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 msgr2=0x7f33d410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:39.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d410aa40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f33c800dfb0 tx=0x7f33c8006ee0).stop 2021-11-20T08:52:39.052 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 shutdown_connections 2021-11-20T08:52:39.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f33c0033c20 0x7f33c0036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:39.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33d406fcd0 0x7f33d410aa40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:39.053 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 --2- 172.21.15.93:0/2966336220 >> [v2:172.21.15.93:6800/3748247363,v1:172.21.15.93:6801/3748247363] conn(0x7f33c002fe00 0x7f33c00322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:39.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.051+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 >> 172.21.15.93:0/2966336220 conn(0x7f33d40f6280 msgr2=0x7f33d40f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:39.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.052+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 shutdown_connections 2021-11-20T08:52:39.054 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.052+0000 7f33da01c700 1 -- 172.21.15.93:0/2966336220 wait complete. 2021-11-20T08:52:39.404 INFO:teuthology.orchestra.run.smithi093.stderr:Deploying grafana service with default placement... 2021-11-20T08:52:39.404 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2021-11-20T08:52:39.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: cluster 2021-11-20T08:52:38.039784+0000 mon.smithi093 (mon 2021-11-20T08:52:39.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: .0) 67 : cluster [DBG] mgrmap e9: smithi093.qdikkm(active, starting, since 0.0526981s) 2021-11-20T08:52:39.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.040905+0000 mon.smithi093 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi093"}]: dispatch 2021-11-20T08:52:39.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.042718+0000 mon.smithi093 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr metadata", "who": "smithi093.qdikkm", "id": "smithi093.qdikkm"}]: dispatch 2021-11-20T08:52:39.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.042919+0000 mon.smithi093 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T08:52:39.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.043066+0000 mon.smithi093 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T08:52:39.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.043212+0000 mon.smithi093 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T08:52:39.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: cluster 2021-11-20T08:52:38.056353+0000 mon.smithi093 (mon.0) 73 : cluster [INF] Manager daemon smithi093.qdikkm is now available 2021-11-20T08:52:39.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.156645+0000 mon.smithi093 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:39.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.157942+0000 mon.smithi093 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:39.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:39.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: 2021-11-20T08:52:38.178123+0000 mon.smithi093 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:52:39.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: audit 2021-11-20T08:52:38.182190+0000 mon.smithi093 (mon 2021-11-20T08:52:39.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:39 smithi093 conmon[33522]: .0) 77 : audit [INF] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:52:39.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.983+0000 7f702101a700 1 Processor -- start 2021-11-20T08:52:39.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.983+0000 7f702101a700 1 -- start start 2021-11-20T08:52:39.985 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.983+0000 7f702101a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c0fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:39.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.983+0000 7f702101a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c0fcb20 con 0x7f701c0fc210 2021-11-20T08:52:39.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f701ad9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c0fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:39.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f701ad9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c0fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42296/0 (socket says 172.21.15.93:42296) 2021-11-20T08:52:39.986 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f701ad9d700 1 -- 172.21.15.93:0/894326425 learned_addr learned my addr 172.21.15.93:0/894326425 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:39.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f701ad9d700 1 -- 172.21.15.93:0/894326425 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c0fcc60 con 0x7f701c0fc210 2021-11-20T08:52:39.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f701ad9d700 1 --2- 172.21.15.93:0/894326425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c0fc630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f701000b5f0 tx=0x7f7010006ee0).ready entity=mon.0 client_cookie=1b4765e9ebca9146 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:39.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f7019d9b700 1 -- 172.21.15.93:0/894326425 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f701000d600 con 0x7f701c0fc210 2021-11-20T08:52:39.987 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f7019d9b700 1 -- 172.21.15.93:0/894326425 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f701000dc10 con 0x7f701c0fc210 2021-11-20T08:52:39.988 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.984+0000 7f7019d9b700 1 -- 172.21.15.93:0/894326425 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7010011cd0 con 0x7f701c0fc210 2021-11-20T08:52:39.988 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.985+0000 7f702101a700 1 -- 172.21.15.93:0/894326425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 msgr2=0x7f701c0fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:39.988 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.985+0000 7f702101a700 1 --2- 172.21.15.93:0/894326425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c0fc630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f701000b5f0 tx=0x7f7010006ee0).stop 2021-11-20T08:52:39.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.985+0000 7f702101a700 1 -- 172.21.15.93:0/894326425 shutdown_connections 2021-11-20T08:52:39.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.985+0000 7f702101a700 1 --2- 172.21.15.93:0/894326425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c0fc630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:39.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.985+0000 7f702101a700 1 -- 172.21.15.93:0/894326425 >> 172.21.15.93:0/894326425 conn(0x7f701c0f5a80 msgr2=0x7f701c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:39.989 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.986+0000 7f702101a700 1 -- 172.21.15.93:0/894326425 shutdown_connections 2021-11-20T08:52:39.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.986+0000 7f702101a700 1 -- 172.21.15.93:0/894326425 wait complete. 2021-11-20T08:52:39.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.986+0000 7f702101a700 1 Processor -- start 2021-11-20T08:52:39.990 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.986+0000 7f702101a700 1 -- start start 2021-11-20T08:52:39.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.986+0000 7f702101a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:39.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.986+0000 7f702101a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c10d0c0 con 0x7f701c0fc210 2021-11-20T08:52:39.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f701ad9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:39.993 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f701ad9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c10cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42298/0 (socket says 172.21.15.93:42298) 2021-11-20T08:52:39.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f701ad9d700 1 -- 172.21.15.93:0/4291631342 learned_addr learned my addr 172.21.15.93:0/4291631342 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:39.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f701ad9d700 1 -- 172.21.15.93:0/4291631342 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701000b200 con 0x7f701c0fc210 2021-11-20T08:52:39.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f701ad9d700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c10cbd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7010006cb0 tx=0x7f7010006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:39.994 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f70037fe700 1 -- 172.21.15.93:0/4291631342 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7010011df0 con 0x7f701c0fc210 2021-11-20T08:52:39.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c10d2c0 con 0x7f701c0fc210 2021-11-20T08:52:39.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f70037fe700 1 -- 172.21.15.93:0/4291631342 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f701001b8c0 con 0x7f701c0fc210 2021-11-20T08:52:39.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f70037fe700 1 -- 172.21.15.93:0/4291631342 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f701001a930 con 0x7f701c0fc210 2021-11-20T08:52:39.995 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.987+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c10d6d0 con 0x7f701c0fc210 2021-11-20T08:52:39.996 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.988+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c045ad0 con 0x7f701c0fc210 2021-11-20T08:52:39.996 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.989+0000 7f70037fe700 1 -- 172.21.15.93:0/4291631342 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f701000f070 con 0x7f701c0fc210 2021-11-20T08:52:39.996 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.989+0000 7f70037fe700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f700802fd10 0x7f70080321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:39.996 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.989+0000 7f70037fe700 1 -- 172.21.15.93:0/4291631342 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7010040e10 con 0x7f701c0fc210 2021-11-20T08:52:39.996 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.989+0000 7f701a59c700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f700802fd10 0x7f70080321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:39.997 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.990+0000 7f701a59c700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f700802fd10 0x7f70080321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f700c0096a0 tx=0x7f700c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:39.997 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:39.993+0000 7f70037fe700 1 -- 172.21.15.93:0/4291631342 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7010010360 con 0x7f701c0fc210 2021-11-20T08:52:40.193 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.194+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 --> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f701c10dfb0 con 0x7f700802fd10 2021-11-20T08:52:40.197 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.198+0000 7f70037fe700 1 -- 172.21.15.93:0/4291631342 <== mgr.14142 v2:172.21.15.93:6800/3294416137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f701c10dfb0 con 0x7f700802fd10 2021-11-20T08:52:40.197 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Scheduled grafana update... 2021-11-20T08:52:40.199 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f700802fd10 msgr2=0x7f70080321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:40.200 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f700802fd10 0x7f70080321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f700c0096a0 tx=0x7f700c006b40).stop 2021-11-20T08:52:40.200 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 msgr2=0x7f701c10cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:40.200 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c10cbd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7010006cb0 tx=0x7f7010006ee0).stop 2021-11-20T08:52:40.200 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 shutdown_connections 2021-11-20T08:52:40.201 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f700802fd10 0x7f70080321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:40.201 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 --2- 172.21.15.93:0/4291631342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f701c0fc210 0x7f701c10cbd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:40.201 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 >> 172.21.15.93:0/4291631342 conn(0x7f701c0f5a80 msgr2=0x7f701c0f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:40.201 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 shutdown_connections 2021-11-20T08:52:40.202 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:40.200+0000 7f702101a700 1 -- 172.21.15.93:0/4291631342 wait complete. 2021-11-20T08:52:40.283 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:40 smithi093 conmon[33522]: cluster 2021-11-20T08:52:39.046925+0000 mon.smithi093 (mon.0) 78 : cluster [DBG] mgrmap e10: smithi093.qdikkm(active, since 1.05984s) 2021-11-20T08:52:40.283 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:40 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:39.047952+0000 mgr.smithi093.qdikkm (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2021-11-20T08:52:40.284 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:40 smithi093 conmon[33522]: audit 2021-11-20T08:52:39.050093+0000 mon.smithi093 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:40.284 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:40 smithi093 conmon[33522]: audit 2021-11-20T08:52:39.854099+0000 mon.smithi093 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:40.694 INFO:teuthology.orchestra.run.smithi093.stderr:Deploying node-exporter service with default placement... 2021-11-20T08:52:40.694 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2021-11-20T08:52:41.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.288+0000 7fb4d3a1c700 1 Processor -- start 2021-11-20T08:52:41.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.288+0000 7fb4d3a1c700 1 -- start start 2021-11-20T08:52:41.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.288+0000 7fb4d3a1c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:41.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.288+0000 7fb4d3a1c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0fb480 con 0x7fb4cc0faae0 2021-11-20T08:52:41.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4d17b8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:41.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4d17b8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42302/0 (socket says 172.21.15.93:42302) 2021-11-20T08:52:41.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4d17b8700 1 -- 172.21.15.93:0/1698756765 learned_addr learned my addr 172.21.15.93:0/1698756765 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:41.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4d17b8700 1 -- 172.21.15.93:0/1698756765 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0fb5c0 con 0x7fb4cc0faae0 2021-11-20T08:52:41.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4d17b8700 1 --2- 172.21.15.93:0/1698756765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc0faf00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb4c800b5f0 tx=0x7fb4c8006ee0).ready entity=mon.0 client_cookie=5d9bd8f64f14415f server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:41.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4bffff700 1 -- 172.21.15.93:0/1698756765 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4c800d600 con 0x7fb4cc0faae0 2021-11-20T08:52:41.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4bffff700 1 -- 172.21.15.93:0/1698756765 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4c800dc10 con 0x7fb4cc0faae0 2021-11-20T08:52:41.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.289+0000 7fb4bffff700 1 -- 172.21.15.93:0/1698756765 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4c8011cd0 con 0x7fb4cc0faae0 2021-11-20T08:52:41.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.290+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/1698756765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 msgr2=0x7fb4cc0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:41.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.290+0000 7fb4d3a1c700 1 --2- 172.21.15.93:0/1698756765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc0faf00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb4c800b5f0 tx=0x7fb4c8006ee0).stop 2021-11-20T08:52:41.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.290+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/1698756765 shutdown_connections 2021-11-20T08:52:41.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.290+0000 7fb4d3a1c700 1 --2- 172.21.15.93:0/1698756765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc0faf00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:41.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.290+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/1698756765 >> 172.21.15.93:0/1698756765 conn(0x7fb4cc0f6280 msgr2=0x7fb4cc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:41.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.290+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/1698756765 shutdown_connections 2021-11-20T08:52:41.297 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.290+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/1698756765 wait complete. 2021-11-20T08:52:41.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d3a1c700 1 Processor -- start 2021-11-20T08:52:41.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d3a1c700 1 -- start start 2021-11-20T08:52:41.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d3a1c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc108880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:41.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d3a1c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc108d70 con 0x7fb4cc0faae0 2021-11-20T08:52:41.298 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d17b8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:41.299 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d17b8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42304/0 (socket says 172.21.15.93:42304) 2021-11-20T08:52:41.299 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d17b8700 1 -- 172.21.15.93:0/3134424383 learned_addr learned my addr 172.21.15.93:0/3134424383 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:41.299 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d17b8700 1 -- 172.21.15.93:0/3134424383 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c800b200 con 0x7fb4cc0faae0 2021-11-20T08:52:41.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.291+0000 7fb4d17b8700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc108880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb4c800dfe0 tx=0x7fb4c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:41.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.292+0000 7fb4bdffb700 1 -- 172.21.15.93:0/3134424383 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4c8011df0 con 0x7fb4cc0faae0 2021-11-20T08:52:41.300 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.292+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc108f70 con 0x7fb4cc0faae0 2021-11-20T08:52:41.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.292+0000 7fb4bdffb700 1 -- 172.21.15.93:0/3134424383 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4c801b8c0 con 0x7fb4cc0faae0 2021-11-20T08:52:41.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.292+0000 7fb4bdffb700 1 -- 172.21.15.93:0/3134424383 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4c801a840 con 0x7fb4cc0faae0 2021-11-20T08:52:41.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.292+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc109380 con 0x7fb4cc0faae0 2021-11-20T08:52:41.301 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.292+0000 7fb4bdffb700 1 -- 172.21.15.93:0/3134424383 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb4c800f070 con 0x7fb4cc0faae0 2021-11-20T08:52:41.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.293+0000 7fb4bdffb700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fb4b40320a0 0x7fb4b4034560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:41.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.293+0000 7fb4bdffb700 1 -- 172.21.15.93:0/3134424383 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb4c80410d0 con 0x7fb4cc0faae0 2021-11-20T08:52:41.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.293+0000 7fb4d0fb7700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fb4b40320a0 0x7fb4b4034560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:41.302 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.293+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc059100 con 0x7fb4cc0faae0 2021-11-20T08:52:41.303 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.293+0000 7fb4d0fb7700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fb4b40320a0 0x7fb4b4034560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb4c00096a0 tx=0x7fb4c0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:41.303 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.297+0000 7fb4bdffb700 1 -- 172.21.15.93:0/3134424383 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4c8016070 con 0x7fb4cc0faae0 2021-11-20T08:52:41.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:41 smithi093 conmon[33522]: audit 2021-11-20T08:52:40.195557+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:52:41.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:41 smithi093 conmon[33522]: mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:41.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:41 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:40.196112+0000 mgr.smithi093.qdikkm (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2021-11-20T08:52:41.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:41 smithi093 conmon[33522]: audit 2021-11-20T08:52:40.198743+0000 mon.smithi093 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:41.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:41 smithi093 conmon[33522]: cluster 2021-11-20T08:52:40.855556+0000 mon.smithi093 (mon.0) 82 : cluster [DBG] mgrmap e11: smithi093.qdikkm(active, since 2s) 2021-11-20T08:52:41.474 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.475+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 --> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4cc102500 con 0x7fb4b40320a0 2021-11-20T08:52:41.477 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.478+0000 7fb4bdffb700 1 -- 172.21.15.93:0/3134424383 <== mgr.14142 v2:172.21.15.93:6800/3294416137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fb4cc102500 con 0x7fb4b40320a0 2021-11-20T08:52:41.477 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2021-11-20T08:52:41.480 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fb4b40320a0 msgr2=0x7fb4b4034560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:41.480 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fb4b40320a0 0x7fb4b4034560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb4c00096a0 tx=0x7fb4c0006b40).stop 2021-11-20T08:52:41.481 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 msgr2=0x7fb4cc108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:41.481 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc108880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb4c800dfe0 tx=0x7fb4c8006ee0).stop 2021-11-20T08:52:41.481 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 shutdown_connections 2021-11-20T08:52:41.481 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fb4b40320a0 0x7fb4b4034560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:41.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 --2- 172.21.15.93:0/3134424383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4cc0faae0 0x7fb4cc108880 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:41.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 >> 172.21.15.93:0/3134424383 conn(0x7fb4cc0f6280 msgr2=0x7fb4cc0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:41.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 shutdown_connections 2021-11-20T08:52:41.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:41.480+0000 7fb4d3a1c700 1 -- 172.21.15.93:0/3134424383 wait complete. 2021-11-20T08:52:41.782 INFO:teuthology.orchestra.run.smithi093.stderr:Deploying alertmanager service with default placement... 2021-11-20T08:52:41.782 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2021-11-20T08:52:42.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.247+0000 7faf485d0700 1 Processor -- start 2021-11-20T08:52:42.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.247+0000 7faf485d0700 1 -- start start 2021-11-20T08:52:42.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.247+0000 7faf485d0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf40100c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:42.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.247+0000 7faf485d0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf400fc680 con 0x7faf4006fc70 2021-11-20T08:52:42.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4636c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf40100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:42.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4636c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf40100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42308/0 (socket says 172.21.15.93:42308) 2021-11-20T08:52:42.253 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4636c700 1 -- 172.21.15.93:0/67197764 learned_addr learned my addr 172.21.15.93:0/67197764 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:42.253 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4636c700 1 -- 172.21.15.93:0/67197764 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf401010f0 con 0x7faf4006fc70 2021-11-20T08:52:42.253 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4636c700 1 --2- 172.21.15.93:0/67197764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf40100c00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7faf34009960 tx=0x7faf340092d0).ready entity=mon.0 client_cookie=9c5c3208e0437d88 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:42.253 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4536a700 1 -- 172.21.15.93:0/67197764 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf3400b9c0 con 0x7faf4006fc70 2021-11-20T08:52:42.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4536a700 1 -- 172.21.15.93:0/67197764 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf34011400 con 0x7faf4006fc70 2021-11-20T08:52:42.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.248+0000 7faf4536a700 1 -- 172.21.15.93:0/67197764 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf34010580 con 0x7faf4006fc70 2021-11-20T08:52:42.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 -- 172.21.15.93:0/67197764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 msgr2=0x7faf40100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:42.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 --2- 172.21.15.93:0/67197764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf40100c00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7faf34009960 tx=0x7faf340092d0).stop 2021-11-20T08:52:42.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 -- 172.21.15.93:0/67197764 shutdown_connections 2021-11-20T08:52:42.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 --2- 172.21.15.93:0/67197764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf40100c00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:42.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 -- 172.21.15.93:0/67197764 >> 172.21.15.93:0/67197764 conn(0x7faf400f6280 msgr2=0x7faf400f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:42.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 -- 172.21.15.93:0/67197764 shutdown_connections 2021-11-20T08:52:42.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 -- 172.21.15.93:0/67197764 wait complete. 2021-11-20T08:52:42.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 Processor -- start 2021-11-20T08:52:42.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.249+0000 7faf485d0700 1 -- start start 2021-11-20T08:52:42.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf485d0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf4010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:42.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf485d0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4010d100 con 0x7faf4006fc70 2021-11-20T08:52:42.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf4636c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf4010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:42.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf4636c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf4010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42310/0 (socket says 172.21.15.93:42310) 2021-11-20T08:52:42.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf4636c700 1 -- 172.21.15.93:0/1206783984 learned_addr learned my addr 172.21.15.93:0/1206783984 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:42.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf4636c700 1 -- 172.21.15.93:0/1206783984 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf34009570 con 0x7faf4006fc70 2021-11-20T08:52:42.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf4636c700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf4010cc10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7faf34006b80 tx=0x7faf340092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:42.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf3effd700 1 -- 172.21.15.93:0/1206783984 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf34010840 con 0x7faf4006fc70 2021-11-20T08:52:42.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf3effd700 1 -- 172.21.15.93:0/1206783984 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf34010e50 con 0x7faf4006fc70 2021-11-20T08:52:42.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4010d300 con 0x7faf4006fc70 2021-11-20T08:52:42.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf3effd700 1 -- 172.21.15.93:0/1206783984 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf3401acb0 con 0x7faf4006fc70 2021-11-20T08:52:42.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.250+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4010d710 con 0x7faf4006fc70 2021-11-20T08:52:42.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.251+0000 7faf3effd700 1 -- 172.21.15.93:0/1206783984 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7faf3400f070 con 0x7faf4006fc70 2021-11-20T08:52:42.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.251+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf40045ad0 con 0x7faf4006fc70 2021-11-20T08:52:42.259 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.251+0000 7faf3effd700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7faf2402feb0 0x7faf24032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:42.260 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.251+0000 7faf3effd700 1 -- 172.21.15.93:0/1206783984 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7faf34041a60 con 0x7faf4006fc70 2021-11-20T08:52:42.260 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.252+0000 7faf45b6b700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7faf2402feb0 0x7faf24032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:42.260 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.252+0000 7faf45b6b700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7faf2402feb0 0x7faf24032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7faf2c0096a0 tx=0x7faf2c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:42.261 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.256+0000 7faf3effd700 1 -- 172.21.15.93:0/1206783984 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf340109a0 con 0x7faf4006fc70 2021-11-20T08:52:42.432 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.433+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 --> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7faf4010dff0 con 0x7faf2402feb0 2021-11-20T08:52:42.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.437+0000 7faf3effd700 1 -- 172.21.15.93:0/1206783984 <== mgr.14142 v2:172.21.15.93:6800/3294416137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7faf4010dff0 con 0x7faf2402feb0 2021-11-20T08:52:42.436 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2021-11-20T08:52:42.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7faf2402feb0 msgr2=0x7faf24032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:42.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7faf2402feb0 0x7faf24032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7faf2c0096a0 tx=0x7faf2c006b40).stop 2021-11-20T08:52:42.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 msgr2=0x7faf4010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:42.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf4010cc10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7faf34006b80 tx=0x7faf340092d0).stop 2021-11-20T08:52:42.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 shutdown_connections 2021-11-20T08:52:42.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7faf2402feb0 0x7faf24032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:42.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 --2- 172.21.15.93:0/1206783984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faf4006fc70 0x7faf4010cc10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:42.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 >> 172.21.15.93:0/1206783984 conn(0x7faf400f6280 msgr2=0x7faf400f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:42.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 shutdown_connections 2021-11-20T08:52:42.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:42.439+0000 7faf485d0700 1 -- 172.21.15.93:0/1206783984 wait complete. 2021-11-20T08:52:42.599 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: audit 2021-11-20T08:52:41.476755+0000 2021-11-20T08:52:42.600 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:42.600 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:41.477432+0000 mgr.smithi093.qdikkm 2021-11-20T08:52:42.600 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2021-11-20T08:52:42.600 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:52:42.601 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: 08:52:41.479370+0000 mon.smithi093 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:42.601 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: audit 2021-11-20T08:52: 2021-11-20T08:52:42.601 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: 42.437485+0000 mon.smithi093 (mon.0) 84 : audit [INF] 2021-11-20T08:52:42.601 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:42 smithi093 conmon[33522]: from='mgr.14142 172.21.15.93:0/629581900' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:42.817 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2021-11-20T08:52:43.312 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.309+0000 7f4bab81a700 1 Processor -- start 2021-11-20T08:52:43.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.309+0000 7f4bab81a700 1 -- start start 2021-11-20T08:52:43.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.309+0000 7f4bab81a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba4070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:43.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.309+0000 7f4bab81a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba40720f0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.313 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4ba95b6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:43.314 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4ba95b6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba4070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42314/0 (socket says 172.21.15.93:42314) 2021-11-20T08:52:43.314 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4ba95b6700 1 -- 172.21.15.93:0/521535611 learned_addr learned my addr 172.21.15.93:0/521535611 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:43.314 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4ba95b6700 1 -- 172.21.15.93:0/521535611 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4070760 con 0x7f4ba4071cd0 2021-11-20T08:52:43.314 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4ba95b6700 1 --2- 172.21.15.93:0/521535611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba4070270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f4ba0009960 tx=0x7f4ba00092d0).ready entity=mon.0 client_cookie=cdfaa213cb66798e server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:43.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4b93fff700 1 -- 172.21.15.93:0/521535611 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ba000b9c0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4b93fff700 1 -- 172.21.15.93:0/521535611 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ba0011400 con 0x7f4ba4071cd0 2021-11-20T08:52:43.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.310+0000 7f4b93fff700 1 -- 172.21.15.93:0/521535611 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ba0010580 con 0x7f4ba4071cd0 2021-11-20T08:52:43.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 -- 172.21.15.93:0/521535611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 msgr2=0x7f4ba4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:43.315 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 --2- 172.21.15.93:0/521535611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba4070270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f4ba0009960 tx=0x7f4ba00092d0).stop 2021-11-20T08:52:43.316 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 -- 172.21.15.93:0/521535611 shutdown_connections 2021-11-20T08:52:43.316 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 --2- 172.21.15.93:0/521535611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba4070270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:43.316 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 -- 172.21.15.93:0/521535611 >> 172.21.15.93:0/521535611 conn(0x7f4ba40f73e0 msgr2=0x7f4ba40f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:43.316 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 -- 172.21.15.93:0/521535611 shutdown_connections 2021-11-20T08:52:43.317 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 -- 172.21.15.93:0/521535611 wait complete. 2021-11-20T08:52:43.317 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 Processor -- start 2021-11-20T08:52:43.317 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.311+0000 7f4bab81a700 1 -- start start 2021-11-20T08:52:43.317 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4bab81a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba410cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:43.318 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4bab81a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba410d0d0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.318 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4ba95b6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba410cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:43.318 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4ba95b6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba410cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42316/0 (socket says 172.21.15.93:42316) 2021-11-20T08:52:43.318 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4ba95b6700 1 -- 172.21.15.93:0/3500899134 learned_addr learned my addr 172.21.15.93:0/3500899134 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:43.318 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4ba95b6700 1 -- 172.21.15.93:0/3500899134 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba0009570 con 0x7f4ba4071cd0 2021-11-20T08:52:43.319 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4ba95b6700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba410cbe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4ba0009f20 tx=0x7f4ba00092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:43.319 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4b91ffb700 1 -- 172.21.15.93:0/3500899134 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ba00103a0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.319 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba410d2d0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.319 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4b91ffb700 1 -- 172.21.15.93:0/3500899134 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ba0010ca0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4b91ffb700 1 -- 172.21.15.93:0/3500899134 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ba001ada0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.312+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba410d6e0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.313+0000 7f4b91ffb700 1 -- 172.21.15.93:0/3500899134 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4ba000f070 con 0x7f4ba4071cd0 2021-11-20T08:52:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.313+0000 7f4b91ffb700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f4b8c02feb0 0x7f4b8c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.313+0000 7f4b91ffb700 1 -- 172.21.15.93:0/3500899134 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4ba0042cb0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.321 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.313+0000 7f4ba8db5700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f4b8c02feb0 0x7f4b8c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:43.321 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.313+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba4059100 con 0x7f4ba4071cd0 2021-11-20T08:52:43.321 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.314+0000 7f4ba8db5700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f4b8c02feb0 0x7f4b8c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4b940096a0 tx=0x7f4b94006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:43.322 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.318+0000 7f4b91ffb700 1 -- 172.21.15.93:0/3500899134 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ba001e070 con 0x7f4ba4071cd0 2021-11-20T08:52:43.464 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.465+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4ba41069c0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.467 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.468+0000 7f4b91ffb700 1 -- 172.21.15.93:0/3500899134 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4ba0023cb0 con 0x7f4ba4071cd0 2021-11-20T08:52:43.471 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f4b8c02feb0 msgr2=0x7f4b8c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:43.471 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f4b8c02feb0 0x7f4b8c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4b940096a0 tx=0x7f4b94006b40).stop 2021-11-20T08:52:43.471 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 msgr2=0x7f4ba410cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:43.471 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba410cbe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4ba0009f20 tx=0x7f4ba00092d0).stop 2021-11-20T08:52:43.472 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 shutdown_connections 2021-11-20T08:52:43.472 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f4b8c02feb0 0x7f4b8c032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:43.472 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 --2- 172.21.15.93:0/3500899134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ba4071cd0 0x7f4ba410cbe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:43.472 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.470+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 >> 172.21.15.93:0/3500899134 conn(0x7f4ba40f73e0 msgr2=0x7f4ba40f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:43.473 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.471+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 shutdown_connections 2021-11-20T08:52:43.473 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:43.471+0000 7f4bab81a700 1 -- 172.21.15.93:0/3500899134 wait complete. 2021-11-20T08:52:43.632 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:43 smithi093 conmon[33522]: audit 2021-11-20T08:52:42.434829+0000 mgr.smithi093.qdikkm (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:43.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:43 smithi093 conmon[33522]: cephadm 2021-11-20T08:52:43.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:43 smithi093 conmon[33522]: 2021-11-20T08:52:42.435351+0000 mgr.smithi093.qdikkm (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2021-11-20T08:52:43.633 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:43 smithi093 conmon[33522]: audit 2021-11-20T08:52:43.469475+0000 mon.smithi093 (mon.0) 85 : audit [INF] from='client.? 172.21.15.93:0/3500899134' entity='client.admin' 2021-11-20T08:52:43.852 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2021-11-20T08:52:44.283 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.280+0000 7f7f944f7700 1 Processor -- start 2021-11-20T08:52:44.284 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.280+0000 7f7f944f7700 1 -- start start 2021-11-20T08:52:44.284 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f944f7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c100be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:44.285 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f944f7700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c0fc660 con 0x7f7f8c06fc90 2021-11-20T08:52:44.285 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f92293700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:44.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f92293700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c100be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42320/0 (socket says 172.21.15.93:42320) 2021-11-20T08:52:44.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f92293700 1 -- 172.21.15.93:0/1073378459 learned_addr learned my addr 172.21.15.93:0/1073378459 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:44.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f92293700 1 -- 172.21.15.93:0/1073378459 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c1010d0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f92293700 1 --2- 172.21.15.93:0/1073378459 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c100be0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7f8800b5f0 tx=0x7f7f88006ee0).ready entity=mon.0 client_cookie=a20df18bfaa529e3 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:44.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f91291700 1 -- 172.21.15.93:0/1073378459 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f8800d600 con 0x7f7f8c06fc90 2021-11-20T08:52:44.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f91291700 1 -- 172.21.15.93:0/1073378459 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f8800dc10 con 0x7f7f8c06fc90 2021-11-20T08:52:44.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.281+0000 7f7f91291700 1 -- 172.21.15.93:0/1073378459 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f88011cd0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.282+0000 7f7f944f7700 1 -- 172.21.15.93:0/1073378459 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 msgr2=0x7f7f8c100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:44.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.282+0000 7f7f944f7700 1 --2- 172.21.15.93:0/1073378459 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c100be0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7f8800b5f0 tx=0x7f7f88006ee0).stop 2021-11-20T08:52:44.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.282+0000 7f7f944f7700 1 -- 172.21.15.93:0/1073378459 shutdown_connections 2021-11-20T08:52:44.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.282+0000 7f7f944f7700 1 --2- 172.21.15.93:0/1073378459 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c100be0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:44.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.282+0000 7f7f944f7700 1 -- 172.21.15.93:0/1073378459 >> 172.21.15.93:0/1073378459 conn(0x7f7f8c0f6280 msgr2=0x7f7f8c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:44.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.282+0000 7f7f944f7700 1 -- 172.21.15.93:0/1073378459 shutdown_connections 2021-11-20T08:52:44.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.282+0000 7f7f944f7700 1 -- 172.21.15.93:0/1073378459 wait complete. 2021-11-20T08:52:44.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f944f7700 1 Processor -- start 2021-11-20T08:52:44.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f944f7700 1 -- start start 2021-11-20T08:52:44.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f944f7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c10cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:44.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f944f7700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c10d0e0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f92293700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c10cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:44.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f92293700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c10cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42322/0 (socket says 172.21.15.93:42322) 2021-11-20T08:52:44.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f92293700 1 -- 172.21.15.93:0/3804114293 learned_addr learned my addr 172.21.15.93:0/3804114293 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:44.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f92293700 1 -- 172.21.15.93:0/3804114293 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8800b200 con 0x7f7f8c06fc90 2021-11-20T08:52:44.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f92293700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c10cbf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7f8800b1d0 tx=0x7f7f88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:44.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f7affd700 1 -- 172.21.15.93:0/3804114293 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f88011df0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f7affd700 1 -- 172.21.15.93:0/3804114293 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f8801b8c0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c10d2e0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.283+0000 7f7f7affd700 1 -- 172.21.15.93:0/3804114293 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f8801a7b0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.284+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c10d6f0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.284+0000 7f7f7affd700 1 -- 172.21.15.93:0/3804114293 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7f8800f070 con 0x7f7f8c06fc90 2021-11-20T08:52:44.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.284+0000 7f7f7affd700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f7f7402fe60 0x7f7f74032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:44.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.284+0000 7f7f7affd700 1 -- 172.21.15.93:0/3804114293 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7f880422e0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.284+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c004fa0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.286+0000 7f7f91a92700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f7f7402fe60 0x7f7f74032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:44.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.286+0000 7f7f91a92700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f7f7402fe60 0x7f7f74032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7f7c0096a0 tx=0x7f7f7c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:44.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.289+0000 7f7f7affd700 1 -- 172.21.15.93:0/3804114293 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f88016020 con 0x7f7f8c06fc90 2021-11-20T08:52:44.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.439+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7f6c005bf0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.442+0000 7f7f7affd700 1 -- 172.21.15.93:0/3804114293 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7f880226d0 con 0x7f7f8c06fc90 2021-11-20T08:52:44.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.443+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f7f7402fe60 msgr2=0x7f7f74032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:44.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.443+0000 7f7f944f7700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f7f7402fe60 0x7f7f74032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7f7c0096a0 tx=0x7f7f7c006b40).stop 2021-11-20T08:52:44.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.443+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 msgr2=0x7f7f8c10cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:44.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.443+0000 7f7f944f7700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c10cbf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7f8800b1d0 tx=0x7f7f88006ee0).stop 2021-11-20T08:52:44.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.444+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 shutdown_connections 2021-11-20T08:52:44.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.444+0000 7f7f944f7700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f7f7402fe60 0x7f7f74032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:44.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.444+0000 7f7f944f7700 1 --2- 172.21.15.93:0/3804114293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f8c06fc90 0x7f7f8c10cbf0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:44.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.444+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 >> 172.21.15.93:0/3804114293 conn(0x7f7f8c0f6280 msgr2=0x7f7f8c0f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:44.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.444+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 shutdown_connections 2021-11-20T08:52:44.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:44.444+0000 7f7f944f7700 1 -- 172.21.15.93:0/3804114293 wait complete. 2021-11-20T08:52:44.786 INFO:teuthology.orchestra.run.smithi093.stderr:Enabling the dashboard module... 2021-11-20T08:52:44.786 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2021-11-20T08:52:45.247 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.245+0000 7f5656ece700 1 Processor -- start 2021-11-20T08:52:45.247 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.246+0000 7f5656ece700 1 -- start start 2021-11-20T08:52:45.248 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.246+0000 7f5656ece700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56500facc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:45.248 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.246+0000 7f5656ece700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56500fb1b0 con 0x7f56500fa8a0 2021-11-20T08:52:45.248 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.246+0000 7f5655ecc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56500facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:45.248 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.246+0000 7f5655ecc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56500facc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42326/0 (socket says 172.21.15.93:42326) 2021-11-20T08:52:45.249 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.246+0000 7f5655ecc700 1 -- 172.21.15.93:0/1808108517 learned_addr learned my addr 172.21.15.93:0/1808108517 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:45.249 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.246+0000 7f5655ecc700 1 -- 172.21.15.93:0/1808108517 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56500fb2f0 con 0x7f56500fa8a0 2021-11-20T08:52:45.249 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5655ecc700 1 --2- 172.21.15.93:0/1808108517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56500facc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f564000b5f0 tx=0x7f5640006ee0).ready entity=mon.0 client_cookie=fe0f0f65253f3a94 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:45.249 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5654eca700 1 -- 172.21.15.93:0/1808108517 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564000d600 con 0x7f56500fa8a0 2021-11-20T08:52:45.249 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5654eca700 1 -- 172.21.15.93:0/1808108517 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f564000dc10 con 0x7f56500fa8a0 2021-11-20T08:52:45.250 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5654eca700 1 -- 172.21.15.93:0/1808108517 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5640011cd0 con 0x7f56500fa8a0 2021-11-20T08:52:45.250 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5656ece700 1 -- 172.21.15.93:0/1808108517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 msgr2=0x7f56500facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:45.250 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5656ece700 1 --2- 172.21.15.93:0/1808108517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56500facc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f564000b5f0 tx=0x7f5640006ee0).stop 2021-11-20T08:52:45.250 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5656ece700 1 -- 172.21.15.93:0/1808108517 shutdown_connections 2021-11-20T08:52:45.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5656ece700 1 --2- 172.21.15.93:0/1808108517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56500facc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:45.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.247+0000 7f5656ece700 1 -- 172.21.15.93:0/1808108517 >> 172.21.15.93:0/1808108517 conn(0x7f56500f60e0 msgr2=0x7f56500f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:45.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5656ece700 1 -- 172.21.15.93:0/1808108517 shutdown_connections 2021-11-20T08:52:45.251 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5656ece700 1 -- 172.21.15.93:0/1808108517 wait complete. 2021-11-20T08:52:45.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5656ece700 1 Processor -- start 2021-11-20T08:52:45.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5656ece700 1 -- start start 2021-11-20T08:52:45.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5656ece700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56501096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:45.252 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5656ece700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650109bc0 con 0x7f56500fa8a0 2021-11-20T08:52:45.254 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5655ecc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56501096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:45.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5655ecc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56501096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42328/0 (socket says 172.21.15.93:42328) 2021-11-20T08:52:45.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.248+0000 7f5655ecc700 1 -- 172.21.15.93:0/4284907392 learned_addr learned my addr 172.21.15.93:0/4284907392 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:45.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.249+0000 7f5655ecc700 1 -- 172.21.15.93:0/4284907392 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564000b200 con 0x7f56500fa8a0 2021-11-20T08:52:45.255 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.249+0000 7f5655ecc700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56501096d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f564000e040 tx=0x7f5640006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:45.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.249+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564001b410 con 0x7f56500fa8a0 2021-11-20T08:52:45.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.249+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f564001ba20 con 0x7f56500fa8a0 2021-11-20T08:52:45.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.249+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564001a910 con 0x7f56500fa8a0 2021-11-20T08:52:45.256 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.249+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5650109dc0 con 0x7f56500fa8a0 2021-11-20T08:52:45.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.249+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565010a1d0 con 0x7f56500fa8a0 2021-11-20T08:52:45.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.250+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f564000f070 con 0x7f56500fa8a0 2021-11-20T08:52:45.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.250+0000 7f563e7fc700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f563802feb0 0x7f5638032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:45.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.250+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f564001bb90 con 0x7f56500fa8a0 2021-11-20T08:52:45.257 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.250+0000 7f56556cb700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f563802feb0 0x7f5638032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:45.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.250+0000 7f56556cb700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f563802feb0 0x7f5638032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f564c0096a0 tx=0x7f564c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:45.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.251+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5650059100 con 0x7f56500fa8a0 2021-11-20T08:52:45.258 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.255+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5640024410 con 0x7f56500fa8a0 2021-11-20T08:52:45.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:45.442+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f5650045ad0 con 0x7f56500fa8a0 2021-11-20T08:52:45.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:45 smithi093 conmon[33522]: audit 2021-11-20T08:52:44.442708+0000 mon.smithi093 (mon. 2021-11-20T08:52:45.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:45 smithi093 conmon[33522]: 0) 86 : audit [INF] from='client.? 172.21.15.93:0/3804114293' entity='client.admin' 2021-11-20T08:52:46.444 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.444+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f564001b570 con 0x7f56500fa8a0 2021-11-20T08:52:46.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.446+0000 7f563e7fc700 1 -- 172.21.15.93:0/4284907392 <== mon.0 v2:172.21.15.93:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f564002d5c0 con 0x7f56500fa8a0 2021-11-20T08:52:46.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f563802feb0 msgr2=0x7f5638032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:46.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f563802feb0 0x7f5638032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f564c0096a0 tx=0x7f564c006b40).stop 2021-11-20T08:52:46.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 msgr2=0x7f56501096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:46.449 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56501096d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f564000e040 tx=0x7f5640006ee0).stop 2021-11-20T08:52:46.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 shutdown_connections 2021-11-20T08:52:46.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7f563802feb0 0x7f5638032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:46.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 --2- 172.21.15.93:0/4284907392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56500fa8a0 0x7f56501096d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:46.450 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 >> 172.21.15.93:0/4284907392 conn(0x7f56500f60e0 msgr2=0x7f56500f7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:46.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 shutdown_connections 2021-11-20T08:52:46.451 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:46.448+0000 7f5656ece700 1 -- 172.21.15.93:0/4284907392 wait complete. 2021-11-20T08:52:46.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:46 smithi093 conmon[33522]: audit 2021-11-20T08:52:45.443410+0000 mon.smithi093 (mon. 2021-11-20T08:52:46.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:46 smithi093 conmon[33522]: 0) 87 : audit [INF] from='client.? 172.21.15.93:0/4284907392' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-11-20T08:52:46.925 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T08:52:47.433 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.433+0000 7fc1550c2700 1 Processor -- start 2021-11-20T08:52:47.434 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.433+0000 7fc1550c2700 1 -- start start 2021-11-20T08:52:47.434 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.433+0000 7fc1550c2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15006fc50 0x7fc150100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:47.434 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.433+0000 7fc1550c2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1500fc6a0 con 0x7fc15006fc50 2021-11-20T08:52:47.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14ed9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15006fc50 0x7fc150100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:47.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14ed9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15006fc50 0x7fc150100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42342/0 (socket says 172.21.15.93:42342) 2021-11-20T08:52:47.435 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14ed9d700 1 -- 172.21.15.93:0/807503240 learned_addr learned my addr 172.21.15.93:0/807503240 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:47.437 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14ed9d700 1 -- 172.21.15.93:0/807503240 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150101110 con 0x7fc15006fc50 2021-11-20T08:52:47.437 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14ed9d700 1 --2- 172.21.15.93:0/807503240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15006fc50 0x7fc150100c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc14400b5f0 tx=0x7fc144006ee0).ready entity=mon.0 client_cookie=2968a44804510296 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:47.438 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14dd9b700 1 -- 172.21.15.93:0/807503240 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14400d600 con 0x7fc15006fc50 2021-11-20T08:52:47.438 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14dd9b700 1 -- 172.21.15.93:0/807503240 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc14400dc10 con 0x7fc15006fc50 2021-11-20T08:52:47.438 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.434+0000 7fc14dd9b700 1 -- 172.21.15.93:0/807503240 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc144011cd0 con 0x7fc15006fc50 2021-11-20T08:52:47.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.435+0000 7fc1550c2700 1 -- 172.21.15.93:0/807503240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15006fc50 msgr2=0x7fc150100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:47.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.435+0000 7fc1550c2700 1 --2- 172.21.15.93:0/807503240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15006fc50 0x7fc150100c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc14400b5f0 tx=0x7fc144006ee0).stop 2021-11-20T08:52:47.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.435+0000 7fc1550c2700 1 -- 172.21.15.93:0/807503240 shutdown_connections 2021-11-20T08:52:47.439 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.435+0000 7fc1550c2700 1 --2- 172.21.15.93:0/807503240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15006fc50 0x7fc150100c20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:47.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.435+0000 7fc1550c2700 1 -- 172.21.15.93:0/807503240 >> 172.21.15.93:0/807503240 conn(0x7fc1500f6280 msgr2=0x7fc1500f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:47.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.436+0000 7fc1550c2700 1 -- 172.21.15.93:0/807503240 shutdown_connections 2021-11-20T08:52:47.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.436+0000 7fc1550c2700 1 -- 172.21.15.93:0/807503240 wait complete. 2021-11-20T08:52:47.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.436+0000 7fc1550c2700 1 Processor -- start 2021-11-20T08:52:47.440 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.436+0000 7fc1550c2700 1 -- start start 2021-11-20T08:52:47.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.437+0000 7fc1550c2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15010ce00 0x7fc15010d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:47.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.437+0000 7fc1550c2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15010d730 con 0x7fc15010ce00 2021-11-20T08:52:47.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.437+0000 7fc14ed9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15010ce00 0x7fc15010d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:47.441 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.437+0000 7fc14ed9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15010ce00 0x7fc15010d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42344/0 (socket says 172.21.15.93:42344) 2021-11-20T08:52:47.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.437+0000 7fc14ed9d700 1 -- 172.21.15.93:0/548163962 learned_addr learned my addr 172.21.15.93:0/548163962 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:47.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.437+0000 7fc14ed9d700 1 -- 172.21.15.93:0/548163962 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14400b200 con 0x7fc15010ce00 2021-11-20T08:52:47.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.437+0000 7fc14ed9d700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15010ce00 0x7fc15010d240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc144013000 tx=0x7fc144006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:47.442 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.438+0000 7fc1377fe700 1 -- 172.21.15.93:0/548163962 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc144011dd0 con 0x7fc15010ce00 2021-11-20T08:52:47.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.438+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15010d930 con 0x7fc15010ce00 2021-11-20T08:52:47.443 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.438+0000 7fc1377fe700 1 -- 172.21.15.93:0/548163962 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc14401b8c0 con 0x7fc15010ce00 2021-11-20T08:52:47.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.438+0000 7fc1377fe700 1 -- 172.21.15.93:0/548163962 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14401a780 con 0x7fc15010ce00 2021-11-20T08:52:47.445 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.438+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15010e320 con 0x7fc15010ce00 2021-11-20T08:52:47.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.438+0000 7fc1377fe700 1 -- 172.21.15.93:0/548163962 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc14400f070 con 0x7fc15010ce00 2021-11-20T08:52:47.446 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.439+0000 7fc1377fe700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 0x7fc13c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:47.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.439+0000 7fc1377fe700 1 -- 172.21.15.93:0/548163962 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc144040f30 con 0x7fc15010ce00 2021-11-20T08:52:47.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.439+0000 7fc14e59c700 1 -- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 msgr2=0x7fc13c0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3294416137 2021-11-20T08:52:47.447 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.439+0000 7fc14e59c700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 0x7fc13c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T08:52:47.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.439+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc150045ad0 con 0x7fc15010ce00 2021-11-20T08:52:47.448 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.443+0000 7fc1377fe700 1 -- 172.21.15.93:0/548163962 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc144016070 con 0x7fc15010ce00 2021-11-20T08:52:47.638 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.638+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc150106710 con 0x7fc15010ce00 2021-11-20T08:52:47.639 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.639+0000 7fc1377fe700 1 -- 172.21.15.93:0/548163962 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7fc144010750 con 0x7fc15010ce00 2021-11-20T08:52:47.639 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: { 2021-11-20T08:52:47.639 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "epoch": 12, 2021-11-20T08:52:47.640 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "available": true, 2021-11-20T08:52:47.640 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "active_name": "smithi093.qdikkm", 2021-11-20T08:52:47.640 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T08:52:47.640 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: } 2021-11-20T08:52:47.640 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.640+0000 7fc14e59c700 1 -- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 msgr2=0x7fc13c0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3294416137 2021-11-20T08:52:47.641 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.640+0000 7fc14e59c700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 0x7fc13c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T08:52:47.643 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 msgr2=0x7fc13c0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T08:52:47.643 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 0x7fc13c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:47.643 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15010ce00 msgr2=0x7fc15010d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:47.643 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15010ce00 0x7fc15010d240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc144013000 tx=0x7fc144006ee0).stop 2021-11-20T08:52:47.644 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 shutdown_connections 2021-11-20T08:52:47.644 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fc13c02ff00 0x7fc13c0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:47.644 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 --2- 172.21.15.93:0/548163962 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc15010ce00 0x7fc15010d240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:47.644 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 >> 172.21.15.93:0/548163962 conn(0x7fc1500f6280 msgr2=0x7fc1500f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:47.645 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 shutdown_connections 2021-11-20T08:52:47.645 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:47.641+0000 7fc1550c2700 1 -- 172.21.15.93:0/548163962 wait complete. 2021-11-20T08:52:47.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:47 smithi093 conmon[33522]: audit 2021-11-20T08:52:46.446992+0000 mon.smithi093 (mon.0) 88 : audit [INF] from='client.? 172.21.15.93:0/4284907392' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-11-20T08:52:47.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:47 smithi093 conmon[33522]: cluster 2021-11-20T08:52:46.447063+0000 mon.smithi093 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi093.qdikkm(active, since 8s) 2021-11-20T08:52:48.064 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for the mgr to restart... 2021-11-20T08:52:48.065 INFO:teuthology.orchestra.run.smithi093.stderr:Waiting for mgr epoch 12... 2021-11-20T08:52:48.065 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T08:52:48.553 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.551+0000 7fd7de0c0700 1 Processor -- start 2021-11-20T08:52:48.553 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.551+0000 7fd7de0c0700 1 -- start start 2021-11-20T08:52:48.554 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.551+0000 7fd7de0c0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d80fa750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:48.554 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.551+0000 7fd7de0c0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d80fac40 con 0x7fd7d80fa330 2021-11-20T08:52:48.554 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.551+0000 7fd7d77fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d80fa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:48.554 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.551+0000 7fd7d77fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d80fa750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42352/0 (socket says 172.21.15.93:42352) 2021-11-20T08:52:48.555 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.551+0000 7fd7d77fe700 1 -- 172.21.15.93:0/848368353 learned_addr learned my addr 172.21.15.93:0/848368353 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:48.555 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.552+0000 7fd7d77fe700 1 -- 172.21.15.93:0/848368353 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d80fad80 con 0x7fd7d80fa330 2021-11-20T08:52:48.555 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.552+0000 7fd7d77fe700 1 --2- 172.21.15.93:0/848368353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d80fa750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd7c800b5f0 tx=0x7fd7c8006ee0).ready entity=mon.0 client_cookie=23ed05292c282100 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:48.555 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.552+0000 7fd7d6ffd700 1 -- 172.21.15.93:0/848368353 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c800d600 con 0x7fd7d80fa330 2021-11-20T08:52:48.556 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.552+0000 7fd7d6ffd700 1 -- 172.21.15.93:0/848368353 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7c800dc10 con 0x7fd7d80fa330 2021-11-20T08:52:48.556 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.552+0000 7fd7d6ffd700 1 -- 172.21.15.93:0/848368353 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c8011cd0 con 0x7fd7d80fa330 2021-11-20T08:52:48.556 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 -- 172.21.15.93:0/848368353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 msgr2=0x7fd7d80fa750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:48.557 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 --2- 172.21.15.93:0/848368353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d80fa750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd7c800b5f0 tx=0x7fd7c8006ee0).stop 2021-11-20T08:52:48.557 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 -- 172.21.15.93:0/848368353 shutdown_connections 2021-11-20T08:52:48.557 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 --2- 172.21.15.93:0/848368353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d80fa750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:48.558 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 -- 172.21.15.93:0/848368353 >> 172.21.15.93:0/848368353 conn(0x7fd7d80f55a0 msgr2=0x7fd7d80f79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:48.558 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 -- 172.21.15.93:0/848368353 shutdown_connections 2021-11-20T08:52:48.558 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 -- 172.21.15.93:0/848368353 wait complete. 2021-11-20T08:52:48.558 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.553+0000 7fd7de0c0700 1 Processor -- start 2021-11-20T08:52:48.559 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7de0c0700 1 -- start start 2021-11-20T08:52:48.559 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7de0c0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d810d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:48.559 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7de0c0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d80f7440 con 0x7fd7d80fa330 2021-11-20T08:52:48.560 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7d77fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d810d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:48.560 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7d77fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d810d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42354/0 (socket says 172.21.15.93:42354) 2021-11-20T08:52:48.561 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7d77fe700 1 -- 172.21.15.93:0/3586700365 learned_addr learned my addr 172.21.15.93:0/3586700365 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:48.562 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7d77fe700 1 -- 172.21.15.93:0/3586700365 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c800b200 con 0x7fd7d80fa330 2021-11-20T08:52:48.562 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.554+0000 7fd7d77fe700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d810d6b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd7c800e040 tx=0x7fd7c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:48.562 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.555+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c801b410 con 0x7fd7d80fa330 2021-11-20T08:52:48.563 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.555+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7c801ba20 con 0x7fd7d80fa330 2021-11-20T08:52:48.563 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.555+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d80f7640 con 0x7fd7d80fa330 2021-11-20T08:52:48.563 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.555+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c801a910 con 0x7fd7d80fa330 2021-11-20T08:52:48.565 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.555+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d810dd60 con 0x7fd7d80fa330 2021-11-20T08:52:48.565 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.555+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd7c800f070 con 0x7fd7d80fa330 2021-11-20T08:52:48.566 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.556+0000 7fd7d4ff9700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 0x7fd7b8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:48.566 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.556+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd7c8042150 con 0x7fd7d80fa330 2021-11-20T08:52:48.566 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.556+0000 7fd7ceffd700 1 -- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 msgr2=0x7fd7b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3294416137 2021-11-20T08:52:48.567 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.556+0000 7fd7ceffd700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 0x7fd7b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T08:52:48.568 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.556+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 --> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7d8059100 con 0x7fd7b802feb0 2021-11-20T08:52:48.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:48 smithi093 conmon[33522]: audit 2021-11-20T08:52:47.639885+0000 mon.smithi093 (mon.0) 2021-11-20T08:52:48.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:48 smithi093 conmon[33522]: 90 : audit [DBG] from='client.? 172.21.15.93:0/548163962' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T08:52:48.756 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.757+0000 7fd7ceffd700 1 -- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 msgr2=0x7fd7b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3294416137 2021-11-20T08:52:48.756 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:48.757+0000 7fd7ceffd700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 0x7fd7b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T08:52:49.157 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:49.158+0000 7fd7ceffd700 1 -- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 msgr2=0x7fd7b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3294416137 2021-11-20T08:52:49.157 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:49.158+0000 7fd7ceffd700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 0x7fd7b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T08:52:49.958 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:49.959+0000 7fd7ceffd700 1 -- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 msgr2=0x7fd7b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.93:6800/3294416137 2021-11-20T08:52:49.958 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:49.959+0000 7fd7ceffd700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 0x7fd7b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T08:52:51.478 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:51.478+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fd7c8024930 con 0x7fd7d80fa330 2021-11-20T08:52:51.479 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:51.478+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 msgr2=0x7fd7b8032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T08:52:51.480 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:51.478+0000 7fd7d4ff9700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 0x7fd7b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:51.727 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:51 smithi093 conmon[33522]: cluster 2021-11-20T08:52:51.424642+0000 2021-11-20T08:52:51.727 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:51 smithi093 conmon[33522]: mon.smithi093 (mon.0) 91 : cluster [INF] Active manager daemon smithi093.qdikkm restarted 2021-11-20T08:52:51.727 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:51 smithi093 conmon[33522]: cluster 2021-11-20T08:52:51.425025+0000 mon.smithi093 (mon.0) 92 : cluster [INF] Activating manager daemon smithi093.qdikkm 2021-11-20T08:52:51.728 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:51 smithi093 conmon[33522]: cluster 2021-11-20T08:52:51.429209+0000 mon.smithi093 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-11-20T08:52:52.486 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.486+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fd7c801b740 con 0x7fd7d80fa330 2021-11-20T08:52:52.487 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.486+0000 7fd7d4ff9700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd7b8033990 0x7fd7b8035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:52.487 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.486+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7d8059100 con 0x7fd7b8033990 2021-11-20T08:52:52.487 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.487+0000 7fd7ceffd700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd7b8033990 0x7fd7b8035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:52.488 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.487+0000 7fd7ceffd700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd7b8033990 0x7fd7b8035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd7c00036d0 tx=0x7fd7c0006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:52.488 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.488+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fd7d8059100 con 0x7fd7b8033990 2021-11-20T08:52:52.496 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.494+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd7d8045ad0 con 0x7fd7b8033990 2021-11-20T08:52:52.496 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.494+0000 7fd7d4ff9700 1 -- 172.21.15.93:0/3586700365 <== mgr.14162 v2:172.21.15.93:6800/3953495879 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fd7d8045ad0 con 0x7fd7b8033990 2021-11-20T08:52:52.496 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: { 2021-11-20T08:52:52.497 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2021-11-20T08:52:52.497 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T08:52:52.497 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: } 2021-11-20T08:52:52.497 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd7b8033990 msgr2=0x7fd7b8035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:52.498 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd7b8033990 0x7fd7b8035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd7c00036d0 tx=0x7fd7c0006ca0).stop 2021-11-20T08:52:52.498 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 msgr2=0x7fd7d810d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:52.498 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d810d6b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd7c800e040 tx=0x7fd7c8006ee0).stop 2021-11-20T08:52:52.499 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 shutdown_connections 2021-11-20T08:52:52.499 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd7b8033990 0x7fd7b8035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:52.500 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7d80fa330 0x7fd7d810d6b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:52.500 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 --2- 172.21.15.93:0/3586700365 >> [v2:172.21.15.93:6800/3294416137,v1:172.21.15.93:6801/3294416137] conn(0x7fd7b802feb0 0x7fd7b8032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:52:52.500 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 >> 172.21.15.93:0/3586700365 conn(0x7fd7d80f55a0 msgr2=0x7fd7d8100890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:52.501 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 shutdown_connections 2021-11-20T08:52:52.501 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:52.495+0000 7fd7de0c0700 1 -- 172.21.15.93:0/3586700365 wait complete. 2021-11-20T08:52:52.910 INFO:teuthology.orchestra.run.smithi093.stderr:mgr epoch 12 is available 2021-11-20T08:52:52.911 INFO:teuthology.orchestra.run.smithi093.stderr:Generating a dashboard self-signed certificate... 2021-11-20T08:52:52.911 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2021-11-20T08:52:52.954 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: cluster 2021-11-20T08:52:51.480374+0000 mon.smithi093 (mon 2021-11-20T08:52:52.954 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: .0) 94 : cluster [DBG] mgrmap e13: smithi093.qdikkm(active, starting, since 0.0554305s) 2021-11-20T08:52:52.954 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11-20T08:52:51.483067+0000 mon.smithi093 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi093"}]: dispatch 2021-11-20T08:52:52.955 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11-20T08:52:51.483648+0000 mon.smithi093 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr metadata", "who": "smithi093.qdikkm", "id": "smithi093.qdikkm"}]: dispatch 2021-11-20T08:52:52.955 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11-20T08:52:51.484271+0000 mon.smithi093 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T08:52:52.955 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11-20T08:52:51. 2021-11-20T08:52:52.956 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: 484395+0000 mon.smithi093 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T08:52:52.956 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11-20T08:52:51.484519+0000 2021-11-20T08:52:52.956 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: mon.smithi093 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T08:52:52.956 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: cluster 2021-11-20T08:52 2021-11-20T08:52:52.957 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: :51.498255+0000 mon.smithi093 (mon.0) 100 : cluster 2021-11-20T08:52:52.958 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: [INF] Manager daemon smithi093.qdikkm is now available 2021-11-20T08:52:52.958 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11-20T08:52:52.959 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: 2021-11-20T08:52:51.594291+0000 mon.smithi093 (mon. 2021-11-20T08:52:52.959 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: 0) 101 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:52.959 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11-20 2021-11-20T08:52:52.959 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: T08:52:51.595842+0000 mon.smithi093 (mon 2021-11-20T08:52:52.959 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: .0) 102 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:52:52.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:52:52.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: -20T08:52:51.613067+0000 mon.smithi093 ( 2021-11-20T08:52:52.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:52:52.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:52:52.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: -20T08:52:51.620186+0000 2021-11-20T08:52:52.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: mon.smithi093 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:52:52.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:52 smithi093 conmon[33522]: 2021-11-20T08:52:53.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f4284eb8700 1 Processor -- start 2021-11-20T08:52:53.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f4284eb8700 1 -- start start 2021-11-20T08:52:53.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f4284eb8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f42800fc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:53.483 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f4284eb8700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42800fca30 con 0x7f42800fc090 2021-11-20T08:52:53.483 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f427e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f42800fc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:53.484 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f427e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f42800fc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42394/0 (socket says 172.21.15.93:42394) 2021-11-20T08:52:53.484 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f427e59c700 1 -- 172.21.15.93:0/2429731748 learned_addr learned my addr 172.21.15.93:0/2429731748 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:53.485 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.476+0000 7f427e59c700 1 -- 172.21.15.93:0/2429731748 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800fcb70 con 0x7f42800fc090 2021-11-20T08:52:53.485 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.477+0000 7f427e59c700 1 --2- 172.21.15.93:0/2429731748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f42800fc4b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f427400b5f0 tx=0x7f4274006ee0).ready entity=mon.0 client_cookie=60d9bb4169c40f60 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:53.485 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.477+0000 7f427d59a700 1 -- 172.21.15.93:0/2429731748 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f427400d600 con 0x7f42800fc090 2021-11-20T08:52:53.486 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.477+0000 7f427d59a700 1 -- 172.21.15.93:0/2429731748 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f427400dc10 con 0x7f42800fc090 2021-11-20T08:52:53.486 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.477+0000 7f427d59a700 1 -- 172.21.15.93:0/2429731748 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274011cd0 con 0x7f42800fc090 2021-11-20T08:52:53.486 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 -- 172.21.15.93:0/2429731748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 msgr2=0x7f42800fc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:53.487 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 --2- 172.21.15.93:0/2429731748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f42800fc4b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f427400b5f0 tx=0x7f4274006ee0).stop 2021-11-20T08:52:53.487 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 -- 172.21.15.93:0/2429731748 shutdown_connections 2021-11-20T08:52:53.488 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 --2- 172.21.15.93:0/2429731748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f42800fc4b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:53.489 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 -- 172.21.15.93:0/2429731748 >> 172.21.15.93:0/2429731748 conn(0x7f42800f59e0 msgr2=0x7f42800f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:53.489 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 -- 172.21.15.93:0/2429731748 shutdown_connections 2021-11-20T08:52:53.490 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 -- 172.21.15.93:0/2429731748 wait complete. 2021-11-20T08:52:53.490 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 Processor -- start 2021-11-20T08:52:53.490 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.478+0000 7f4284eb8700 1 -- start start 2021-11-20T08:52:53.491 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f4284eb8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f4280100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:53.491 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f4284eb8700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42800ff300 con 0x7f42800fc090 2021-11-20T08:52:53.491 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f427e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f4280100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:53.491 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f427e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f4280100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42396/0 (socket says 172.21.15.93:42396) 2021-11-20T08:52:53.492 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f427e59c700 1 -- 172.21.15.93:0/2159947697 learned_addr learned my addr 172.21.15.93:0/2159947697 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:53.492 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f427e59c700 1 -- 172.21.15.93:0/2159947697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427400b200 con 0x7f42800fc090 2021-11-20T08:52:53.492 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f427e59c700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f4280100e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f4274006cb0 tx=0x7f4274006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:53.492 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f426effd700 1 -- 172.21.15.93:0/2159947697 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274011df0 con 0x7f42800fc090 2021-11-20T08:52:53.493 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42800ff4a0 con 0x7f42800fc090 2021-11-20T08:52:53.493 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42800ff8b0 con 0x7f42800fc090 2021-11-20T08:52:53.493 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f426effd700 1 -- 172.21.15.93:0/2159947697 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42740248c0 con 0x7f42800fc090 2021-11-20T08:52:53.493 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.479+0000 7f426effd700 1 -- 172.21.15.93:0/2159947697 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f427401b8f0 con 0x7f42800fc090 2021-11-20T08:52:53.494 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.480+0000 7f426effd700 1 -- 172.21.15.93:0/2159947697 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f427400f070 con 0x7f42800fc090 2021-11-20T08:52:53.494 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.480+0000 7f426effd700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f426402fdc0 0x7f4264032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:53.494 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.481+0000 7f426effd700 1 -- 172.21.15.93:0/2159947697 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4274041e80 con 0x7f42800fc090 2021-11-20T08:52:53.495 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.481+0000 7f427dd9b700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f426402fdc0 0x7f4264032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:53.495 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.481+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4280059100 con 0x7f42800fc090 2021-11-20T08:52:53.496 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.481+0000 7f427dd9b700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f426402fdc0 0x7f4264032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f42700096a0 tx=0x7f4270006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:53.497 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.485+0000 7f426effd700 1 -- 172.21.15.93:0/2159947697 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4274023370 con 0x7f42800fc090 2021-11-20T08:52:53.649 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.649+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f4280104a10 con 0x7f426402fdc0 2021-11-20T08:52:54.104 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.700+0000 7f426effd700 1 -- 172.21.15.93:0/2159947697 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f4280104a10 con 0x7f426402fdc0 2021-11-20T08:52:54.104 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: Self-signed certificate created 2021-11-20T08:52:54.105 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f426402fdc0 msgr2=0x7f4264032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:54.105 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f426402fdc0 0x7f4264032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f42700096a0 tx=0x7f4270006b40).stop 2021-11-20T08:52:54.105 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 msgr2=0x7f4280100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:54.105 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f4280100e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f4274006cb0 tx=0x7f4274006ee0).stop 2021-11-20T08:52:54.106 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 shutdown_connections 2021-11-20T08:52:54.106 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f426402fdc0 0x7f4264032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:54.107 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 --2- 172.21.15.93:0/2159947697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42800fc090 0x7f4280100e80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:54.107 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 >> 172.21.15.93:0/2159947697 conn(0x7f42800f59e0 msgr2=0x7f42800fa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:54.108 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 shutdown_connections 2021-11-20T08:52:54.108 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:53.702+0000 7f4284eb8700 1 -- 172.21.15.93:0/2159947697 wait complete. 2021-11-20T08:52:54.109 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:53 smithi093 conmon[33522]: audit 2021-11-20T08:52:52.488494+0000 mgr.smithi093.qdikkm (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T08:52:54.109 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:53 smithi093 conmon[33522]: cluster 2021-11-20T08:52:52.489392+0000 mon.smithi093 (mon.0) 105 : cluster [DBG] mgrmap e14: smithi093.qdikkm(active, since 1.06445s) 2021-11-20T08:52:54.109 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:53 smithi093 conmon[33522]: audit 2021-11-20T08:52:53.346982+0000 mon.smithi093 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:54.147 INFO:teuthology.orchestra.run.smithi093.stderr:Creating initial admin user... 2021-11-20T08:52:54.148 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7cwl08h2:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-11-20T08:52:54.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:54 smithi093 conmon[33522]: audit 2021-11-20T08:52:52.495644+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:52:54.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:54 smithi093 conmon[33522]: mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T08:52:54.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:54 smithi093 conmon[33522]: audit 2021-11-20T08:52:53.699247+0000 mon.smithi093 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:54.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:54 smithi093 conmon[33522]: audit 2021-11-20T08:52:53.701190+0000 mon.smithi093 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:54.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:54 smithi093 conmon[33522]: cluster 2021-11-20T08:52:54.348429+0000 mon.smithi093 (mon.0) 109 : cluster [DBG] mgrmap e15: smithi093.qdikkm(active, since 2s) 2021-11-20T08:52:55.285 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.085+0000 7f52ef59e700 1 Processor -- start 2021-11-20T08:52:55.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.086+0000 7f52ef59e700 1 -- start start 2021-11-20T08:52:55.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.086+0000 7f52ef59e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f00facc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:55.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.086+0000 7f52ef59e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00fb1b0 con 0x7f52f00fa8a0 2021-11-20T08:52:55.286 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.086+0000 7f52ee59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f00facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:55.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.086+0000 7f52ee59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f00facc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42400/0 (socket says 172.21.15.93:42400) 2021-11-20T08:52:55.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.086+0000 7f52ee59c700 1 -- 172.21.15.93:0/239342515 learned_addr learned my addr 172.21.15.93:0/239342515 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:55.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.086+0000 7f52ee59c700 1 -- 172.21.15.93:0/239342515 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00fb2f0 con 0x7f52f00fa8a0 2021-11-20T08:52:55.287 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ee59c700 1 --2- 172.21.15.93:0/239342515 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f00facc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f52e000b5f0 tx=0x7f52e0006ee0).ready entity=mon.0 client_cookie=3dfbb099b6209d65 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:55.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ed59a700 1 -- 172.21.15.93:0/239342515 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e000d600 con 0x7f52f00fa8a0 2021-11-20T08:52:55.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ed59a700 1 -- 172.21.15.93:0/239342515 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52e000dc10 con 0x7f52f00fa8a0 2021-11-20T08:52:55.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ed59a700 1 -- 172.21.15.93:0/239342515 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e0011cd0 con 0x7f52f00fa8a0 2021-11-20T08:52:55.288 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ef59e700 1 -- 172.21.15.93:0/239342515 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 msgr2=0x7f52f00facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:55.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ef59e700 1 --2- 172.21.15.93:0/239342515 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f00facc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f52e000b5f0 tx=0x7f52e0006ee0).stop 2021-11-20T08:52:55.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ef59e700 1 -- 172.21.15.93:0/239342515 shutdown_connections 2021-11-20T08:52:55.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ef59e700 1 --2- 172.21.15.93:0/239342515 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f00facc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:55.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ef59e700 1 -- 172.21.15.93:0/239342515 >> 172.21.15.93:0/239342515 conn(0x7f52f00f60e0 msgr2=0x7f52f00f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:55.289 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ef59e700 1 -- 172.21.15.93:0/239342515 shutdown_connections 2021-11-20T08:52:55.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.087+0000 7f52ef59e700 1 -- 172.21.15.93:0/239342515 wait complete. 2021-11-20T08:52:55.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ef59e700 1 Processor -- start 2021-11-20T08:52:55.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ef59e700 1 -- start start 2021-11-20T08:52:55.290 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ef59e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f010a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:55.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ef59e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f010ad20 con 0x7f52f00fa8a0 2021-11-20T08:52:55.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ee59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f010a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:55.291 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ee59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f010a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42402/0 (socket says 172.21.15.93:42402) 2021-11-20T08:52:55.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ee59c700 1 -- 172.21.15.93:0/775449211 learned_addr learned my addr 172.21.15.93:0/775449211 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:55.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ee59c700 1 -- 172.21.15.93:0/775449211 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e000b200 con 0x7f52f00fa8a0 2021-11-20T08:52:55.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.088+0000 7f52ee59c700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f010a830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f52e000dfb0 tx=0x7f52e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:55.292 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.089+0000 7f52deffd700 1 -- 172.21.15.93:0/775449211 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e0011df0 con 0x7f52f00fa8a0 2021-11-20T08:52:55.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.089+0000 7f52deffd700 1 -- 172.21.15.93:0/775449211 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52e001b8c0 con 0x7f52f00fa8a0 2021-11-20T08:52:55.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.089+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f010af20 con 0x7f52f00fa8a0 2021-11-20T08:52:55.293 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.089+0000 7f52deffd700 1 -- 172.21.15.93:0/775449211 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e001a8c0 con 0x7f52f00fa8a0 2021-11-20T08:52:55.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.089+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f010b330 con 0x7f52f00fa8a0 2021-11-20T08:52:55.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.089+0000 7f52deffd700 1 -- 172.21.15.93:0/775449211 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f52e000f070 con 0x7f52f00fa8a0 2021-11-20T08:52:55.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.089+0000 7f52deffd700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f52d402ff00 0x7f52d40323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:55.294 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.090+0000 7f52deffd700 1 -- 172.21.15.93:0/775449211 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52e0041110 con 0x7f52f00fa8a0 2021-11-20T08:52:55.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.090+0000 7f52edd9b700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f52d402ff00 0x7f52d40323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:55.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.090+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f0059100 con 0x7f52f00fa8a0 2021-11-20T08:52:55.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.090+0000 7f52edd9b700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f52d402ff00 0x7f52d40323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f52e40096a0 tx=0x7f52e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:55.295 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.094+0000 7f52deffd700 1 -- 172.21.15.93:0/775449211 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52e0016070 con 0x7f52f00fa8a0 2021-11-20T08:52:55.296 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.260+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f52f0045ad0 con 0x7f52d402ff00 2021-11-20T08:52:55.479 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.480+0000 7f52deffd700 1 -- 172.21.15.93:0/775449211 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f52f0045ad0 con 0x7f52d402ff00 2021-11-20T08:52:55.479 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$axleoBHVd1EtRvSzPVWIVuviH.8ekjjNVJvcc3aWuLVVT0f58a9Y2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1637398375, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-11-20T08:52:55.481 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f52d402ff00 msgr2=0x7f52d40323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:55.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f52d402ff00 0x7f52d40323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f52e40096a0 tx=0x7f52e4006b40).stop 2021-11-20T08:52:55.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 msgr2=0x7f52f010a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:55.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f010a830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f52e000dfb0 tx=0x7f52e0006ee0).stop 2021-11-20T08:52:55.482 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 shutdown_connections 2021-11-20T08:52:55.483 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f52d402ff00 0x7f52d40323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:55.483 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 --2- 172.21.15.93:0/775449211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52f00fa8a0 0x7f52f010a830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:55.483 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 >> 172.21.15.93:0/775449211 conn(0x7f52f00f60e0 msgr2=0x7f52f006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:55.483 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 shutdown_connections 2021-11-20T08:52:55.484 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:55.482+0000 7f52ef59e700 1 -- 172.21.15.93:0/775449211 wait complete. 2021-11-20T08:52:55.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:55 smithi093 conmon[33522]: audit 2021-11-20T08:52:53.651312+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:52:55.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:55 smithi093 conmon[33522]: 14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:55.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:55 smithi093 conmon[33522]: audit 2021-11-20T08:52:55.481008+0000 mon.smithi093 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:52:56.187 INFO:teuthology.orchestra.run.smithi093.stderr:Fetching dashboard port number... 2021-11-20T08:52:56.187 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi093 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp1csi2w7k:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpy27msnlh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2021-11-20T08:52:56.632 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.629+0000 7f257f164700 1 Processor -- start 2021-11-20T08:52:56.632 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.629+0000 7f257f164700 1 -- start start 2021-11-20T08:52:56.632 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.629+0000 7f257f164700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:56.633 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.629+0000 7f257f164700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25780fdb30 con 0x7f25780fd220 2021-11-20T08:52:56.633 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.629+0000 7f257cf00700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:56.633 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.629+0000 7f257cf00700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42406/0 (socket says 172.21.15.93:42406) 2021-11-20T08:52:56.634 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.629+0000 7f257cf00700 1 -- 172.21.15.93:0/3170839896 learned_addr learned my addr 172.21.15.93:0/3170839896 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:56.634 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.630+0000 7f257cf00700 1 -- 172.21.15.93:0/3170839896 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25780fdc70 con 0x7f25780fd220 2021-11-20T08:52:56.634 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.630+0000 7f257cf00700 1 --2- 172.21.15.93:0/3170839896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fd640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f256c00b5f0 tx=0x7f256c006ee0).ready entity=mon.0 client_cookie=d66f826d0bfce6f4 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:56.635 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.630+0000 7f25777fe700 1 -- 172.21.15.93:0/3170839896 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256c00d600 con 0x7f25780fd220 2021-11-20T08:52:56.635 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.630+0000 7f25777fe700 1 -- 172.21.15.93:0/3170839896 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f256c00dc10 con 0x7f25780fd220 2021-11-20T08:52:56.635 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.630+0000 7f25777fe700 1 -- 172.21.15.93:0/3170839896 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256c011cd0 con 0x7f25780fd220 2021-11-20T08:52:56.635 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.631+0000 7f257f164700 1 -- 172.21.15.93:0/3170839896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 msgr2=0x7f25780fd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:56.636 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.631+0000 7f257f164700 1 --2- 172.21.15.93:0/3170839896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fd640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f256c00b5f0 tx=0x7f256c006ee0).stop 2021-11-20T08:52:56.636 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.631+0000 7f257f164700 1 -- 172.21.15.93:0/3170839896 shutdown_connections 2021-11-20T08:52:56.636 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.631+0000 7f257f164700 1 --2- 172.21.15.93:0/3170839896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fd640 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:56.636 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.631+0000 7f257f164700 1 -- 172.21.15.93:0/3170839896 >> 172.21.15.93:0/3170839896 conn(0x7f25780f5a80 msgr2=0x7f25780f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:56.637 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.631+0000 7f257f164700 1 -- 172.21.15.93:0/3170839896 shutdown_connections 2021-11-20T08:52:56.637 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.631+0000 7f257f164700 1 -- 172.21.15.93:0/3170839896 wait complete. 2021-11-20T08:52:56.637 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.632+0000 7f257f164700 1 Processor -- start 2021-11-20T08:52:56.637 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.632+0000 7f257f164700 1 -- start start 2021-11-20T08:52:56.637 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.632+0000 7f257f164700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:56.638 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.632+0000 7f257f164700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25780faf70 con 0x7f25780fd220 2021-11-20T08:52:56.638 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.632+0000 7f257cf00700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:56.639 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.632+0000 7f257cf00700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42408/0 (socket says 172.21.15.93:42408) 2021-11-20T08:52:56.639 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.632+0000 7f257cf00700 1 -- 172.21.15.93:0/1481532714 learned_addr learned my addr 172.21.15.93:0/1481532714 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:56.640 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.633+0000 7f257cf00700 1 -- 172.21.15.93:0/1481532714 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c00b200 con 0x7f25780fd220 2021-11-20T08:52:56.640 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.633+0000 7f257cf00700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fcaf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f256c006cb0 tx=0x7f256c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:56.641 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.633+0000 7f25757fa700 1 -- 172.21.15.93:0/1481532714 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256c011df0 con 0x7f25780fd220 2021-11-20T08:52:56.641 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.633+0000 7f25757fa700 1 -- 172.21.15.93:0/1481532714 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f256c01b8c0 con 0x7f25780fd220 2021-11-20T08:52:56.642 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.633+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25780fb110 con 0x7f25780fd220 2021-11-20T08:52:56.642 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.633+0000 7f25757fa700 1 -- 172.21.15.93:0/1481532714 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256c01a930 con 0x7f25780fd220 2021-11-20T08:52:56.644 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.633+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25780fb520 con 0x7f25780fd220 2021-11-20T08:52:56.644 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.634+0000 7f25757fa700 1 -- 172.21.15.93:0/1481532714 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f256c00f070 con 0x7f25780fd220 2021-11-20T08:52:56.644 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.634+0000 7f25757fa700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f255c02ff00 0x7f255c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:56.645 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.634+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2578045ad0 con 0x7f25780fd220 2021-11-20T08:52:56.645 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.634+0000 7f25757fa700 1 -- 172.21.15.93:0/1481532714 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f256c0410f0 con 0x7f25780fd220 2021-11-20T08:52:56.645 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.634+0000 7f2577fff700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f255c02ff00 0x7f255c0323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:56.646 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.635+0000 7f2577fff700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f255c02ff00 0x7f255c0323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f25640096a0 tx=0x7f2564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:56.647 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.639+0000 7f25757fa700 1 -- 172.21.15.93:0/1481532714 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f256c016070 con 0x7f25780fd220 2021-11-20T08:52:56.793 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:56 smithi093 conmon[33522]: audit 2021-11-20T08:52:55.262244+0000 mgr.smithi093.qdikkm 2021-11-20T08:52:56.794 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:56 smithi093 conmon[33522]: (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:52:56.796 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.794+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f25780fa650 con 0x7f25780fd220 2021-11-20T08:52:56.796 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.795+0000 7f25757fa700 1 -- 172.21.15.93:0/1481532714 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f256c024410 con 0x7f25780fd220 2021-11-20T08:52:56.797 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 8443 2021-11-20T08:52:56.797 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.796+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f255c02ff00 msgr2=0x7f255c0323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:56.797 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.796+0000 7f257f164700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f255c02ff00 0x7f255c0323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f25640096a0 tx=0x7f2564006b40).stop 2021-11-20T08:52:56.797 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.796+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 msgr2=0x7f25780fcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:56.798 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.796+0000 7f257f164700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fcaf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f256c006cb0 tx=0x7f256c006ee0).stop 2021-11-20T08:52:56.798 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.797+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 shutdown_connections 2021-11-20T08:52:56.799 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.797+0000 7f257f164700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f255c02ff00 0x7f255c0323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:56.799 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.797+0000 7f257f164700 1 --2- 172.21.15.93:0/1481532714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25780fd220 0x7f25780fcaf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:56.799 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.797+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 >> 172.21.15.93:0/1481532714 conn(0x7f25780f5a80 msgr2=0x7f25780ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:56.800 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.797+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 shutdown_connections 2021-11-20T08:52:56.800 INFO:teuthology.orchestra.run.smithi093.stderr:/usr/bin/ceph: 2021-11-20T08:52:56.797+0000 7f257f164700 1 -- 172.21.15.93:0/1481532714 wait complete. 2021-11-20T08:52:57.122 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T08:52:57.131 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: disabled 2021-11-20T08:52:57.131 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T08:52:57.137 INFO:teuthology.orchestra.run.smithi093.stderr:systemctl: inactive 2021-11-20T08:52:57.138 INFO:teuthology.orchestra.run.smithi093.stderr:firewalld.service is not enabled 2021-11-20T08:52:57.139 INFO:teuthology.orchestra.run.smithi093.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-11-20T08:52:57.140 INFO:teuthology.orchestra.run.smithi093.stderr:Ceph Dashboard is now available at: 2021-11-20T08:52:57.140 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.141 INFO:teuthology.orchestra.run.smithi093.stderr: URL: https://smithi093.front.sepia.ceph.com:8443/ 2021-11-20T08:52:57.141 INFO:teuthology.orchestra.run.smithi093.stderr: User: admin 2021-11-20T08:52:57.141 INFO:teuthology.orchestra.run.smithi093.stderr: Password: pyufidvzhj 2021-11-20T08:52:57.141 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.142 INFO:teuthology.orchestra.run.smithi093.stderr:You can access the Ceph CLI with: 2021-11-20T08:52:57.143 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.143 INFO:teuthology.orchestra.run.smithi093.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-11-20T08:52:57.143 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.143 INFO:teuthology.orchestra.run.smithi093.stderr:Please consider enabling telemetry to help improve Ceph: 2021-11-20T08:52:57.144 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.144 INFO:teuthology.orchestra.run.smithi093.stderr: ceph telemetry on 2021-11-20T08:52:57.144 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.144 INFO:teuthology.orchestra.run.smithi093.stderr:For more information see: 2021-11-20T08:52:57.145 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.145 INFO:teuthology.orchestra.run.smithi093.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2021-11-20T08:52:57.145 INFO:teuthology.orchestra.run.smithi093.stderr: 2021-11-20T08:52:57.145 INFO:teuthology.orchestra.run.smithi093.stderr:Bootstrap complete. 2021-11-20T08:52:57.146 INFO:teuthology.orchestra.run.smithi093.stderr:Releasing lock 140552188062408 on /run/cephadm/11458b34-49df-11ec-8c2c-001a4aab830c.lock 2021-11-20T08:52:57.146 INFO:teuthology.orchestra.run.smithi093.stderr:Lock 140552188062408 released on /run/cephadm/11458b34-49df-11ec-8c2c-001a4aab830c.lock 2021-11-20T08:52:57.175 INFO:tasks.cephadm:Fetching config... 2021-11-20T08:52:57.176 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:52:57.176 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-11-20T08:52:57.193 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-11-20T08:52:57.194 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:52:57.194 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-11-20T08:52:57.250 INFO:tasks.cephadm:Fetching mon keyring... 2021-11-20T08:52:57.250 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:52:57.251 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/keyring of=/dev/stdout 2021-11-20T08:52:57.317 INFO:tasks.cephadm:Fetching pub ssh key... 2021-11-20T08:52:57.318 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:52:57.318 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-11-20T08:52:57.376 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-11-20T08:52:57.377 DEBUG:teuthology.orchestra.run.smithi093:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj3OnmdS6uXgb20MvhfkMc1OUojWrpYTu+JfuMhv50/XZ8QkN8u5yQsEDN4j8nMaxhO/1+uGS9TT+Zf3bKjKnYfYD/5sy8tbFU1hqHUese/3ifHfl7nPuEzxAq/ITrT0dfsZpjDdMzCXAzTHc7Der9TFBcR/n3dpYVIYM23QXvcABYF3kCiTDM7w543xk2K7SqbTXem47lJ7P/lG51hflw5N5dnNBMyA0UcX6aSzw+8JJSXthpQ3CVnU9OgEnxrUzaiqxSj9W53k9r3Ff6DjQRPLlV4UOvSSKbrV3gR9Nw7dzYIYDTCwcgZyQ7O+x7kwk0jZV1Ysru/2Q9b1adKv4QdrqaTfD7ilEQgM42ApF5QwaYEmRDk2p/TH/4fDIW2YLSTjugq6E0/PRhvgLRfRsmtZwOQ2r8lvPx6oLpDTE9dQTJUWYe7KvZSkCfwY66N+II1p51l9BaUAF+ec+AUTlJB+1jAL2KEo6m0lghcz/lNoXDeVEPu0x8rr0KzSsLB+8= ceph-11458b34-49df-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T08:52:57.452 INFO:teuthology.orchestra.run.smithi093.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj3OnmdS6uXgb20MvhfkMc1OUojWrpYTu+JfuMhv50/XZ8QkN8u5yQsEDN4j8nMaxhO/1+uGS9TT+Zf3bKjKnYfYD/5sy8tbFU1hqHUese/3ifHfl7nPuEzxAq/ITrT0dfsZpjDdMzCXAzTHc7Der9TFBcR/n3dpYVIYM23QXvcABYF3kCiTDM7w543xk2K7SqbTXem47lJ7P/lG51hflw5N5dnNBMyA0UcX6aSzw+8JJSXthpQ3CVnU9OgEnxrUzaiqxSj9W53k9r3Ff6DjQRPLlV4UOvSSKbrV3gR9Nw7dzYIYDTCwcgZyQ7O+x7kwk0jZV1Ysru/2Q9b1adKv4QdrqaTfD7ilEQgM42ApF5QwaYEmRDk2p/TH/4fDIW2YLSTjugq6E0/PRhvgLRfRsmtZwOQ2r8lvPx6oLpDTE9dQTJUWYe7KvZSkCfwY66N+II1p51l9BaUAF+ec+AUTlJB+1jAL2KEo6m0lghcz/lNoXDeVEPu0x8rr0KzSsLB+8= ceph-11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:52:57.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj3OnmdS6uXgb20MvhfkMc1OUojWrpYTu+JfuMhv50/XZ8QkN8u5yQsEDN4j8nMaxhO/1+uGS9TT+Zf3bKjKnYfYD/5sy8tbFU1hqHUese/3ifHfl7nPuEzxAq/ITrT0dfsZpjDdMzCXAzTHc7Der9TFBcR/n3dpYVIYM23QXvcABYF3kCiTDM7w543xk2K7SqbTXem47lJ7P/lG51hflw5N5dnNBMyA0UcX6aSzw+8JJSXthpQ3CVnU9OgEnxrUzaiqxSj9W53k9r3Ff6DjQRPLlV4UOvSSKbrV3gR9Nw7dzYIYDTCwcgZyQ7O+x7kwk0jZV1Ysru/2Q9b1adKv4QdrqaTfD7ilEQgM42ApF5QwaYEmRDk2p/TH/4fDIW2YLSTjugq6E0/PRhvgLRfRsmtZwOQ2r8lvPx6oLpDTE9dQTJUWYe7KvZSkCfwY66N+II1p51l9BaUAF+ec+AUTlJB+1jAL2KEo6m0lghcz/lNoXDeVEPu0x8rr0KzSsLB+8= ceph-11458b34-49df-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T08:52:57.508 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj3OnmdS6uXgb20MvhfkMc1OUojWrpYTu+JfuMhv50/XZ8QkN8u5yQsEDN4j8nMaxhO/1+uGS9TT+Zf3bKjKnYfYD/5sy8tbFU1hqHUese/3ifHfl7nPuEzxAq/ITrT0dfsZpjDdMzCXAzTHc7Der9TFBcR/n3dpYVIYM23QXvcABYF3kCiTDM7w543xk2K7SqbTXem47lJ7P/lG51hflw5N5dnNBMyA0UcX6aSzw+8JJSXthpQ3CVnU9OgEnxrUzaiqxSj9W53k9r3Ff6DjQRPLlV4UOvSSKbrV3gR9Nw7dzYIYDTCwcgZyQ7O+x7kwk0jZV1Ysru/2Q9b1adKv4QdrqaTfD7ilEQgM42ApF5QwaYEmRDk2p/TH/4fDIW2YLSTjugq6E0/PRhvgLRfRsmtZwOQ2r8lvPx6oLpDTE9dQTJUWYe7KvZSkCfwY66N+II1p51l9BaUAF+ec+AUTlJB+1jAL2KEo6m0lghcz/lNoXDeVEPu0x8rr0KzSsLB+8= ceph-11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:52:57.521 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-11-20T08:52:57.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:57 smithi093 conmon[33522]: audit 2021-11-20T08:52:56.795910+0000 mon.smithi093 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.93:0/1481532714' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-11-20T08:52:58.826 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:58 smithi093 conmon[33522]: cluster 2021-11-20T08:52:57.495191+0000 mon.smithi093 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi093.qdikkm(active, since 6s) 2021-11-20T08:52:58.826 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:52:58 smithi093 conmon[33522]: 2021-11-20T08:52:59.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 -- 172.21.15.93:0/860222244 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 msgr2=0x7f54000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:59.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 --2- 172.21.15.93:0/860222244 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f54000f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f53f4009fb0 tx=0x7f53f4009b30).stop 2021-11-20T08:52:59.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 -- 172.21.15.93:0/860222244 shutdown_connections 2021-11-20T08:52:59.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 --2- 172.21.15.93:0/860222244 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f54000f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:59.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 -- 172.21.15.93:0/860222244 >> 172.21.15.93:0/860222244 conn(0x7f54000f1c70 msgr2=0x7f54000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:59.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 -- 172.21.15.93:0/860222244 shutdown_connections 2021-11-20T08:52:59.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 -- 172.21.15.93:0/860222244 wait complete. 2021-11-20T08:52:59.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.180+0000 7f5406688700 1 Processor -- start 2021-11-20T08:52:59.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f5406688700 1 -- start start 2021-11-20T08:52:59.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f5406688700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f5400106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:59.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f5406688700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5400106c20 con 0x7f54000f6510 2021-11-20T08:52:59.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f5400106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:59.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f5400106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42418/0 (socket says 172.21.15.93:42418) 2021-11-20T08:52:59.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fffff700 1 -- 172.21.15.93:0/126546663 learned_addr learned my addr 172.21.15.93:0/126546663 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:52:59.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fffff700 1 -- 172.21.15.93:0/126546663 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f4005040 con 0x7f54000f6510 2021-11-20T08:52:59.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fffff700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f5400106730 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f53f4014000 tx=0x7f53f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:59.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fcff9700 1 -- 172.21.15.93:0/126546663 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f40095f0 con 0x7f54000f6510 2021-11-20T08:52:59.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5400106e20 con 0x7f54000f6510 2021-11-20T08:52:59.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fcff9700 1 -- 172.21.15.93:0/126546663 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53f4007e70 con 0x7f54000f6510 2021-11-20T08:52:59.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f53fcff9700 1 -- 172.21.15.93:0/126546663 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f4014ea0 con 0x7f54000f6510 2021-11-20T08:52:59.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.181+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54001072b0 con 0x7f54000f6510 2021-11-20T08:52:59.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.182+0000 7f53fcff9700 1 -- 172.21.15.93:0/126546663 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53f4025070 con 0x7f54000f6510 2021-11-20T08:52:59.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.182+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5400045ad0 con 0x7f54000f6510 2021-11-20T08:52:59.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.182+0000 7f53fcff9700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f53e802ff60 0x7f53e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:52:59.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.182+0000 7f53fcff9700 1 -- 172.21.15.93:0/126546663 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53f40409c0 con 0x7f54000f6510 2021-11-20T08:52:59.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.182+0000 7f53ff7fe700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f53e802ff60 0x7f53e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:52:59.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.183+0000 7f53ff7fe700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f53e802ff60 0x7f53e8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f53f00096a0 tx=0x7f53f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:52:59.189 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.187+0000 7f53fcff9700 1 -- 172.21.15.93:0/126546663 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53f400f710 con 0x7f54000f6510 2021-11-20T08:52:59.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.345+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f54000fbf60 con 0x7f54000f6510 2021-11-20T08:52:59.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.348+0000 7f53fcff9700 1 -- 172.21.15.93:0/126546663 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f53f4025300 con 0x7f54000f6510 2021-11-20T08:52:59.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f53e802ff60 msgr2=0x7f53e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:59.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f53e802ff60 0x7f53e8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f53f00096a0 tx=0x7f53f0006b40).stop 2021-11-20T08:52:59.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 msgr2=0x7f5400106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:52:59.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f5400106730 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f53f4014000 tx=0x7f53f400b040).stop 2021-11-20T08:52:59.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 shutdown_connections 2021-11-20T08:52:59.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f53e802ff60 0x7f53e8032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:59.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 --2- 172.21.15.93:0/126546663 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54000f6510 0x7f5400106730 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:52:59.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 >> 172.21.15.93:0/126546663 conn(0x7f54000f1c70 msgr2=0x7f54000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:52:59.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 shutdown_connections 2021-11-20T08:52:59.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:52:59.350+0000 7f5406688700 1 -- 172.21.15.93:0/126546663 wait complete. 2021-11-20T08:52:59.831 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi102 2021-11-20T08:52:59.831 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:52:59.832 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.conf 2021-11-20T08:52:59.850 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:52:59.851 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T08:52:59.907 INFO:tasks.cephadm:Adding host smithi102 to orchestrator... 2021-11-20T08:52:59.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch host add smithi102 2021-11-20T08:53:00.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:00 smithi093 conmon[33522]: audit 2021-11-20T08:52:59.349089+0000 mon.smithi093 (mon.0) 113 : audit [INF] from='client.? 172.21.15.93:0/126546663' entity='client.admin' 2021-11-20T08:53:01.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:01 smithi093 conmon[33522]: audit 2021-11-20T08:53:00.681705+0000 mon.smithi093 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:01.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:01 smithi093 conmon[33522]: audit 2021-11-20T08:53:01.052017+0000 mon.smithi093 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:01.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:01 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:01.058363+0000 mgr.smithi093.qdikkm (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi093 on smithi093 2021-11-20T08:53:08.021 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:07 smithi093 conmon[33522]: audit 2021-11-20T08:53:06.920566+0000 mon.smithi093 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:08.022 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:07 smithi093 conmon[33522]: audit 2021-11-20T08:53:06.921521+0000 mon.smithi093 (mon.0) 117 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi093", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T08:53:08.022 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:07 smithi093 conmon[33522]: audit 2021-11-20T08:53:06.923157+0000 mon.smithi093 (mon.0) 118 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi093", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T08:53:08.023 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:07 smithi093 conmon[33522]: audit 2021-11-20T08:53:06.923749+0000 mon.smithi093 (mon.0) 119 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:53:08.023 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:07 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:06.924251+0000 mgr.smithi093.qdikkm (mgr.14162) 6 : cephadm [INF] Deploying daemon crash.smithi093 on smithi093 2021-11-20T08:53:10.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: audit 2021-11-20T08:53:08.968185+0000 mon.smithi093 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:10.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: audit 2021-11-20T08:53:09.135160+0000 mon.smithi093 (mon.0) 121 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:10.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: audit 2021-11-20T08:53:09.136974+0000 mon.smithi093 (mon.0) 122 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:10.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: audit 2021-11-20T08:53:09.137333+0000 mon.smithi093 (mon.0) 123 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T08:53:10.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: audit 2021-11-20T08:53:09.137734+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:53:10.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: .14162) 7 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T08:53:10.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: audit 2021-11-20T08:53:09. 2021-11-20T08:53:10.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: 140042+0000 mon.smithi093 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:10.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: cephadm 2021- 2021-11-20T08:53:10.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: 11-20T08:53:09.142861+0000 2021-11-20T08:53:10.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 8 : cephadm [INF] 2021-11-20T08:53:10.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:09 smithi093 conmon[33522]: Deploying daemon grafana.smithi093 on smithi093 2021-11-20T08:53:12.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:12 smithi093 conmon[33522]: cluster 2021-11-20T08:53:11.485310+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:53:12.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:12 smithi093 conmon[33522]: .14162) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:14.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:14 smithi093 conmon[33522]: cluster 2021-11-20T08:53:13.485542+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:53:14.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:14 smithi093 conmon[33522]: 14162) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:15.446 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/3143812803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 msgr2=0x7f5a880f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 --2- 172.21.15.102:0/3143812803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a880f6ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5a84009fb0 tx=0x7f5a84009b30).stop 2021-11-20T08:53:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/3143812803 shutdown_connections 2021-11-20T08:53:15.447 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 --2- 172.21.15.102:0/3143812803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a880f6ea0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:15.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/3143812803 >> 172.21.15.102:0/3143812803 conn(0x7f5a880f02e0 msgr2=0x7f5a880f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:15.448 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/3143812803 shutdown_connections 2021-11-20T08:53:15.449 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/3143812803 wait complete. 2021-11-20T08:53:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 Processor -- start 2021-11-20T08:53:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.443+0000 7f5a8f2cb700 1 -- start start 2021-11-20T08:53:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8f2cb700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a88109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:15.450 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8f2cb700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880f5330 con 0x7f5a880f6a80 2021-11-20T08:53:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8d067700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a88109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8d067700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a88109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54908/0 (socket says 172.21.15.102:54908) 2021-11-20T08:53:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8d067700 1 -- 172.21.15.102:0/4051964380 learned_addr learned my addr 172.21.15.102:0/4051964380 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:15.451 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8d067700 1 -- 172.21.15.102:0/4051964380 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a84005040 con 0x7f5a880f6a80 2021-11-20T08:53:15.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8d067700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a88109410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5a84013510 tx=0x7f5a8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:15.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a7dffb700 1 -- 172.21.15.102:0/4051964380 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a84009320 con 0x7f5a880f6a80 2021-11-20T08:53:15.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a7dffb700 1 -- 172.21.15.102:0/4051964380 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a84013f00 con 0x7f5a880f6a80 2021-11-20T08:53:15.452 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a880f5530 con 0x7f5a880f6a80 2021-11-20T08:53:15.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.444+0000 7f5a7dffb700 1 -- 172.21.15.102:0/4051964380 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a84014dc0 con 0x7f5a880f6a80 2021-11-20T08:53:15.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.445+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a88109ac0 con 0x7f5a880f6a80 2021-11-20T08:53:15.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.445+0000 7f5a7dffb700 1 -- 172.21.15.102:0/4051964380 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a84014420 con 0x7f5a880f6a80 2021-11-20T08:53:15.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.445+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a88045ad0 con 0x7f5a880f6a80 2021-11-20T08:53:15.453 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.445+0000 7f5a7dffb700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5a7402ff60 0x7f5a74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:15.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.445+0000 7f5a7dffb700 1 -- 172.21.15.102:0/4051964380 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a840410e0 con 0x7f5a880f6a80 2021-11-20T08:53:15.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.446+0000 7f5a8c866700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5a7402ff60 0x7f5a74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:15.454 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.446+0000 7f5a8c866700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5a7402ff60 0x7f5a74032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5a780096a0 tx=0x7f5a78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:15.455 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.451+0000 7f5a7dffb700 1 -- 172.21.15.102:0/4051964380 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a840095f0 con 0x7f5a880f6a80 2021-11-20T08:53:15.622 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:15.622+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a880f14c0 con 0x7f5a7402ff60 2021-11-20T08:53:16.315 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi102' 2021-11-20T08:53:16.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.315+0000 7f5a7dffb700 1 -- 172.21.15.102:0/4051964380 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f5a880f14c0 con 0x7f5a7402ff60 2021-11-20T08:53:16.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5a7402ff60 msgr2=0x7f5a74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:16.317 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5a7402ff60 0x7f5a74032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5a780096a0 tx=0x7f5a78006b40).stop 2021-11-20T08:53:16.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 msgr2=0x7f5a88109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:16.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a88109410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5a84013510 tx=0x7f5a8400b040).stop 2021-11-20T08:53:16.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 shutdown_connections 2021-11-20T08:53:16.318 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5a7402ff60 0x7f5a74032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:16.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 --2- 172.21.15.102:0/4051964380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a880f6a80 0x7f5a88109410 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:16.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 >> 172.21.15.102:0/4051964380 conn(0x7f5a880f02e0 msgr2=0x7f5a880f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:16.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 shutdown_connections 2021-11-20T08:53:16.319 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:16.317+0000 7f5a8f2cb700 1 -- 172.21.15.102:0/4051964380 wait complete. 2021-11-20T08:53:16.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch host ls --format=json 2021-11-20T08:53:16.770 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:16 smithi093 conmon[33522]: cluster 2021-11-20T08:53: 2021-11-20T08:53:16.770 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:16 smithi093 conmon[33522]: 15.485801+0000 mgr.smithi093.qdikkm (mgr.14162) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:16.771 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:16 smithi093 conmon[33522]: audit 2021-11-20T08:53:16.315694 2021-11-20T08:53:16.771 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:16 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:17.852 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:17 smithi093 conmon[33522]: audit 2021-11-20T08:53:15.625041+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:53:17.853 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:17 smithi093 conmon[33522]: 12 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:53:17.853 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:17 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:16.316051+0000 mgr.smithi093.qdikkm (mgr.14162) 13 : cephadm [INF] Added host smithi102 2021-11-20T08:53:17.853 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:17 smithi093 conmon[33522]: 2021-11-20T08:53:18.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 -- 172.21.15.102:0/1956338867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 msgr2=0x7fd3b80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:18.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 --2- 172.21.15.102:0/1956338867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b80f4be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fd3b4009fb0 tx=0x7fd3b4009b30).stop 2021-11-20T08:53:18.062 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 -- 172.21.15.102:0/1956338867 shutdown_connections 2021-11-20T08:53:18.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 --2- 172.21.15.102:0/1956338867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b80f4be0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:18.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 -- 172.21.15.102:0/1956338867 >> 172.21.15.102:0/1956338867 conn(0x7fd3b80f02e0 msgr2=0x7fd3b80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:18.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 -- 172.21.15.102:0/1956338867 shutdown_connections 2021-11-20T08:53:18.063 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 -- 172.21.15.102:0/1956338867 wait complete. 2021-11-20T08:53:18.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 Processor -- start 2021-11-20T08:53:18.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 -- start start 2021-11-20T08:53:18.064 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.061+0000 7fd3c01d2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8106bf0 con 0x7fd3b80f47c0 2021-11-20T08:53:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3bdf6e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3bdf6e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54914/0 (socket says 172.21.15.102:54914) 2021-11-20T08:53:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3bdf6e700 1 -- 172.21.15.102:0/3506763500 learned_addr learned my addr 172.21.15.102:0/3506763500 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:18.065 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3bdf6e700 1 -- 172.21.15.102:0/3506763500 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b4005040 con 0x7fd3b80f47c0 2021-11-20T08:53:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3bdf6e700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b8106700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fd3b4013510 tx=0x7fd3b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3ae7fc700 1 -- 172.21.15.102:0/3506763500 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3b4009320 con 0x7fd3b80f47c0 2021-11-20T08:53:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b8106df0 con 0x7fd3b80f47c0 2021-11-20T08:53:18.066 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3ae7fc700 1 -- 172.21.15.102:0/3506763500 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3b4013f00 con 0x7fd3b80f47c0 2021-11-20T08:53:18.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3ae7fc700 1 -- 172.21.15.102:0/3506763500 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3b4014d90 con 0x7fd3b80f47c0 2021-11-20T08:53:18.067 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.062+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b8107280 con 0x7fd3b80f47c0 2021-11-20T08:53:18.068 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.063+0000 7fd3ae7fc700 1 -- 172.21.15.102:0/3506763500 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3b4014420 con 0x7fd3b80f47c0 2021-11-20T08:53:18.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.063+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b8045ad0 con 0x7fd3b80f47c0 2021-11-20T08:53:18.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.063+0000 7fd3ae7fc700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd3a402ffb0 0x7fd3a4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:18.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.063+0000 7fd3ae7fc700 1 -- 172.21.15.102:0/3506763500 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3b40410c0 con 0x7fd3b80f47c0 2021-11-20T08:53:18.069 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.063+0000 7fd3bd76d700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd3a402ffb0 0x7fd3a4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:18.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.064+0000 7fd3bd76d700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd3a402ffb0 0x7fd3a4032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3a80096a0 tx=0x7fd3a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:18.070 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.068+0000 7fd3ae7fc700 1 -- 172.21.15.102:0/3506763500 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3b40095f0 con 0x7fd3b80f47c0 2021-11-20T08:53:18.239 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.239+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3b80f14c0 con 0x7fd3a402ffb0 2021-11-20T08:53:18.240 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.241+0000 7fd3ae7fc700 1 -- 172.21.15.102:0/3506763500 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fd3b80f14c0 con 0x7fd3a402ffb0 2021-11-20T08:53:18.240 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:18.241 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.93", "hostname": "smithi093", "labels": [], "status": ""}, {"addr": "smithi102", "hostname": "smithi102", "labels": [], "status": ""}] 2021-11-20T08:53:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd3a402ffb0 msgr2=0x7fd3a4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd3a402ffb0 0x7fd3a4032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3a80096a0 tx=0x7fd3a8006b40).stop 2021-11-20T08:53:18.242 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 msgr2=0x7fd3b8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b8106700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fd3b4013510 tx=0x7fd3b400b040).stop 2021-11-20T08:53:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 shutdown_connections 2021-11-20T08:53:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd3a402ffb0 0x7fd3a4032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 --2- 172.21.15.102:0/3506763500 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3b80f47c0 0x7fd3b8106700 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 >> 172.21.15.102:0/3506763500 conn(0x7fd3b80f02e0 msgr2=0x7fd3b80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:18.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 shutdown_connections 2021-11-20T08:53:18.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:18.242+0000 7fd3c01d2700 1 -- 172.21.15.102:0/3506763500 wait complete. 2021-11-20T08:53:18.641 INFO:tasks.cephadm:Setting crush tunables to default 2021-11-20T08:53:18.642 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd crush tunables default 2021-11-20T08:53:18.762 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:18 smithi093 conmon[33522]: cluster 2021-11-20T08:53: 2021-11-20T08:53:18.763 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:18 smithi093 conmon[33522]: 17.486195+0000 mgr.smithi093.qdikkm (mgr.14162) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:19.614 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:19 smithi093 conmon[33522]: audit 2021-11-20T08:53:18.241818+0000 mgr.smithi093.qdikkm (mgr.14162) 15 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T08:53:19.615 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:19 smithi093 conmon[33522]: audit 2021-11-20T08:53:19.615 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:19 smithi093 conmon[33522]: 2021-11-20T08:53:19.478354+0000 mon.smithi093 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:20.538 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 -- 172.21.15.93:0/1576321649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 msgr2=0x7f20b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:20.538 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 --2- 172.21.15.93:0/1576321649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b00f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2098009fb0 tx=0x7f2098009b30).stop 2021-11-20T08:53:20.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 -- 172.21.15.93:0/1576321649 shutdown_connections 2021-11-20T08:53:20.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 --2- 172.21.15.93:0/1576321649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b00f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:20.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 -- 172.21.15.93:0/1576321649 >> 172.21.15.93:0/1576321649 conn(0x7f20b00f0b70 msgr2=0x7f20b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:20.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 -- 172.21.15.93:0/1576321649 shutdown_connections 2021-11-20T08:53:20.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 -- 172.21.15.93:0/1576321649 wait complete. 2021-11-20T08:53:20.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 Processor -- start 2021-11-20T08:53:20.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.536+0000 7f20b70ae700 1 -- start start 2021-11-20T08:53:20.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b70ae700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:20.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b70ae700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b0105b20 con 0x7f20b00f5410 2021-11-20T08:53:20.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b4e4a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:20.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b4e4a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42462/0 (socket says 172.21.15.93:42462) 2021-11-20T08:53:20.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b4e4a700 1 -- 172.21.15.93:0/2022970715 learned_addr learned my addr 172.21.15.93:0/2022970715 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:53:20.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b4e4a700 1 -- 172.21.15.93:0/2022970715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2098005040 con 0x7f20b00f5410 2021-11-20T08:53:20.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b4e4a700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b0105630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2098014000 tx=0x7f209800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:20.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20ad7fa700 1 -- 172.21.15.93:0/2022970715 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20980095f0 con 0x7f20b00f5410 2021-11-20T08:53:20.543 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b0105d20 con 0x7f20b00f5410 2021-11-20T08:53:20.543 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20ad7fa700 1 -- 172.21.15.93:0/2022970715 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2098007e70 con 0x7f20b00f5410 2021-11-20T08:53:20.543 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20ad7fa700 1 -- 172.21.15.93:0/2022970715 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2098014ea0 con 0x7f20b00f5410 2021-11-20T08:53:20.543 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.537+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b01061b0 con 0x7f20b00f5410 2021-11-20T08:53:20.544 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.538+0000 7f20ad7fa700 1 -- 172.21.15.93:0/2022970715 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2098027070 con 0x7f20b00f5410 2021-11-20T08:53:20.544 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.538+0000 7f20ad7fa700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f209c02ff60 0x7f209c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:20.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.538+0000 7f20ad7fa700 1 -- 172.21.15.93:0/2022970715 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20980426e0 con 0x7f20b00f5410 2021-11-20T08:53:20.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.538+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b0045ad0 con 0x7f20b00f5410 2021-11-20T08:53:20.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.539+0000 7f20affff700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f209c02ff60 0x7f209c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:20.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.539+0000 7f20affff700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f209c02ff60 0x7f209c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f20a00089e0 tx=0x7f20a0006ec0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:20.547 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.543+0000 7f20ad7fa700 1 -- 172.21.15.93:0/2022970715 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f209801f3c0 con 0x7f20b00f5410 2021-11-20T08:53:20.702 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:20.702+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f20b00faf50 con 0x7f20b00f5410 2021-11-20T08:53:20.851 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:20 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:19.479864+0000 mgr.smithi093.qdikkm (mgr.14162) 16 : cephadm [INF] Deploying daemon node-exporter.smithi093 on smithi093 2021-11-20T08:53:20.852 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:20 smithi093 conmon[33522]: cluster 2021-11-20T08:53:19.486495+0000 mgr.smithi093.qdikkm (mgr.14162) 17 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:21.508 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.508+0000 7f20ad7fa700 1 -- 172.21.15.93:0/2022970715 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f209801e320 con 0x7f20b00f5410 2021-11-20T08:53:21.510 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f209c02ff60 msgr2=0x7f209c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:21.510 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f209c02ff60 0x7f209c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f20a00089e0 tx=0x7f20a0006ec0).stop 2021-11-20T08:53:21.510 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 msgr2=0x7f20b0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:21.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b0105630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2098014000 tx=0x7f209800b040).stop 2021-11-20T08:53:21.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 shutdown_connections 2021-11-20T08:53:21.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f209c02ff60 0x7f209c032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:21.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 --2- 172.21.15.93:0/2022970715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20b00f5410 0x7f20b0105630 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:21.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 >> 172.21.15.93:0/2022970715 conn(0x7f20b00f0b70 msgr2=0x7f20b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:21.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 shutdown_connections 2021-11-20T08:53:21.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:21.510+0000 7f20b70ae700 1 -- 172.21.15.93:0/2022970715 wait complete. 2021-11-20T08:53:21.513 INFO:teuthology.orchestra.run.smithi093.stderr:adjusted tunables profile to default 2021-11-20T08:53:21.797 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:21 smithi093 conmon[33522]: audit 2021-11-20T08:53:20.704669+0000 mon.smithi093 (mon.0) 127 : 2021-11-20T08:53:21.797 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:21 smithi093 conmon[33522]: audit [INF] from='client.? 172.21.15.93:0/2022970715' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-11-20T08:53:21.798 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:21 smithi093 conmon[33522]: cluster 2021-11-20T08:53:21.486714+0000 mgr.smithi093.qdikkm (mgr.14162) 18 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:22.039 INFO:tasks.cephadm:Adding mon.smithi093 on smithi093 2021-11-20T08:53:22.039 INFO:tasks.cephadm:Adding mon.smithi102 on smithi102 2021-11-20T08:53:22.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch apply mon '2;smithi093:172.21.15.93=smithi093;smithi102:172.21.15.102=smithi102' 2021-11-20T08:53:22.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:22 smithi093 conmon[33522]: audit 2021-11-20T08:53:21.510376+0000 mon.smithi093 (mon.0) 128 : audit [INF] from='client.? 172.21.15.93:0/2022970715' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-11-20T08:53:22.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:22 smithi093 conmon[33522]: cluster 2021-11-20T08 2021-11-20T08:53:22.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:22 smithi093 conmon[33522]: :53:21.510469+0000 mon.smithi093 (mon.0) 129 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T08:53:23.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.370+0000 7fed98eaa700 1 -- 172.21.15.102:0/764728184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 msgr2=0x7fed940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:23.372 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.370+0000 7fed98eaa700 1 --2- 172.21.15.102:0/764728184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed940f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fed84009fb0 tx=0x7fed84009b30).stop 2021-11-20T08:53:23.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.370+0000 7fed98eaa700 1 -- 172.21.15.102:0/764728184 shutdown_connections 2021-11-20T08:53:23.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.370+0000 7fed98eaa700 1 --2- 172.21.15.102:0/764728184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed940f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:23.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.370+0000 7fed98eaa700 1 -- 172.21.15.102:0/764728184 >> 172.21.15.102:0/764728184 conn(0x7fed940f1c70 msgr2=0x7fed940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:23.373 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.370+0000 7fed98eaa700 1 -- 172.21.15.102:0/764728184 shutdown_connections 2021-11-20T08:53:23.374 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.370+0000 7fed98eaa700 1 -- 172.21.15.102:0/764728184 wait complete. 2021-11-20T08:53:23.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.371+0000 7fed98eaa700 1 Processor -- start 2021-11-20T08:53:23.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.371+0000 7fed98eaa700 1 -- start start 2021-11-20T08:53:23.377 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.371+0000 7fed98eaa700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed9406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.371+0000 7fed98eaa700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed94069e20 con 0x7fed940f6510 2021-11-20T08:53:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.371+0000 7fed9259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed9406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.371+0000 7fed9259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed9406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54920/0 (socket says 172.21.15.102:54920) 2021-11-20T08:53:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.371+0000 7fed9259c700 1 -- 172.21.15.102:0/659040051 learned_addr learned my addr 172.21.15.102:0/659040051 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:23.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.372+0000 7fed9259c700 1 -- 172.21.15.102:0/659040051 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84005040 con 0x7fed940f6510 2021-11-20T08:53:23.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.372+0000 7fed9259c700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed9406b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fed84013510 tx=0x7fed8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:23.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.372+0000 7fed82ffd700 1 -- 172.21.15.102:0/659040051 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed84009320 con 0x7fed940f6510 2021-11-20T08:53:23.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.372+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed94069fc0 con 0x7fed940f6510 2021-11-20T08:53:23.379 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.372+0000 7fed82ffd700 1 -- 172.21.15.102:0/659040051 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed84013f00 con 0x7fed940f6510 2021-11-20T08:53:23.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.372+0000 7fed82ffd700 1 -- 172.21.15.102:0/659040051 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed84014d90 con 0x7fed940f6510 2021-11-20T08:53:23.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.372+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9406a450 con 0x7fed940f6510 2021-11-20T08:53:23.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.373+0000 7fed82ffd700 1 -- 172.21.15.102:0/659040051 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed840263a0 con 0x7fed940f6510 2021-11-20T08:53:23.380 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.373+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed94059100 con 0x7fed940f6510 2021-11-20T08:53:23.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.373+0000 7fed82ffd700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed7c02ff60 0x7fed7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:23.381 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.373+0000 7fed82ffd700 1 -- 172.21.15.102:0/659040051 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed84041380 con 0x7fed940f6510 2021-11-20T08:53:23.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.373+0000 7fed91d9b700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed7c02ff60 0x7fed7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:23.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.373+0000 7fed91d9b700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed7c02ff60 0x7fed7c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fed880096a0 tx=0x7fed88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:23.382 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.378+0000 7fed82ffd700 1 -- 172.21.15.102:0/659040051 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed8401ccf0 con 0x7fed940f6510 2021-11-20T08:53:23.551 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.552+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi093:172.21.15.93=smithi093;smithi102:172.21.15.102=smithi102", "target": ["mon-mgr", ""]}) v1 -- 0x7fed940f2e20 con 0x7fed7c02ff60 2021-11-20T08:53:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.557+0000 7fed82ffd700 1 -- 172.21.15.102:0/659040051 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fed940f2e20 con 0x7fed7c02ff60 2021-11-20T08:53:23.556 INFO:teuthology.orchestra.run.smithi102.stdout:Scheduled mon update... 2021-11-20T08:53:23.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed7c02ff60 msgr2=0x7fed7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:23.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed7c02ff60 0x7fed7c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fed880096a0 tx=0x7fed88006b40).stop 2021-11-20T08:53:23.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 msgr2=0x7fed9406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:23.558 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed9406b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fed84013510 tx=0x7fed8400b040).stop 2021-11-20T08:53:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 shutdown_connections 2021-11-20T08:53:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed7c02ff60 0x7fed7c032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 --2- 172.21.15.102:0/659040051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed940f6510 0x7fed9406b9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 >> 172.21.15.102:0/659040051 conn(0x7fed940f1c70 msgr2=0x7fed940f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:23.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 shutdown_connections 2021-11-20T08:53:23.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:23.558+0000 7fed98eaa700 1 -- 172.21.15.102:0/659040051 wait complete. 2021-11-20T08:53:23.975 DEBUG:teuthology.orchestra.run.smithi102:mon.smithi102> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi102.service 2021-11-20T08:53:23.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T08:53:23.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T08:53:24.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:-- Logs begin at Sat 2021-11-20 08:41:07 UTC. -- 2021-11-20T08:53:24.210 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:23 smithi093 conmon[33522]: audit 2021-11-20T08:53:22.721695+0000 mon.smithi093 (mon.0) 130 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:24.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:23 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:22 2021-11-20T08:53:24.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:23 smithi093 conmon[33522]: .727099+0000 mgr.smithi093.qdikkm (mgr.14162) 19 : cephadm [INF] Deploying daemon prometheus.smithi093 on smithi093 2021-11-20T08:53:24.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:23 smithi093 conmon[33522]: cluster 2021-11-20T08:53:23.487089+0000 mgr.smithi093.qdikkm (mgr.14162) 20 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:24.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:23 smithi093 conmon[33522]: audit 2021-11-20T08:53:23.558031+0000 mon.smithi093 (mon.0) 131 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:25.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:24 smithi093 conmon[33522]: audit 2021-11-20T08:53:23.554526+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:53:25.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:24 smithi093 conmon[33522]: 21 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi093:172.21.15.93=smithi093;smithi102:172.21.15.102=smithi102", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:53:25.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:24 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:23.555703+0000 mgr.smithi093.qdikkm (mgr.14162) 22 : cephadm [INF] Saving service mon spec with placement smithi093:172.21.15.93=smithi093;smithi102:172.21.15.102=smithi102;count:2 2021-11-20T08:53:25.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 -- 172.21.15.102:0/3879124539 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 msgr2=0x7f95980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:25.521 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 --2- 172.21.15.102:0/3879124539 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f95980f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9594009fb0 tx=0x7f9594009b30).stop 2021-11-20T08:53:25.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 -- 172.21.15.102:0/3879124539 shutdown_connections 2021-11-20T08:53:25.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 --2- 172.21.15.102:0/3879124539 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f95980f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:25.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 -- 172.21.15.102:0/3879124539 >> 172.21.15.102:0/3879124539 conn(0x7f95980f0b70 msgr2=0x7f95980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:25.522 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 -- 172.21.15.102:0/3879124539 shutdown_connections 2021-11-20T08:53:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 -- 172.21.15.102:0/3879124539 wait complete. 2021-11-20T08:53:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.519+0000 7f959f8c3700 1 Processor -- start 2021-11-20T08:53:25.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959f8c3700 1 -- start start 2021-11-20T08:53:25.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959f8c3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f9598109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:25.524 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959f8c3700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9598060240 con 0x7f95980f5410 2021-11-20T08:53:25.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959d65f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f9598109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:25.525 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959d65f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f9598109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54926/0 (socket says 172.21.15.102:54926) 2021-11-20T08:53:25.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959d65f700 1 -- 172.21.15.102:0/2321241189 learned_addr learned my addr 172.21.15.102:0/2321241189 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:25.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959d65f700 1 -- 172.21.15.102:0/2321241189 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9594005040 con 0x7f95980f5410 2021-11-20T08:53:25.526 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959d65f700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f9598109400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9594013510 tx=0x7f959400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f958dffb700 1 -- 172.21.15.102:0/2321241189 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9594009320 con 0x7f95980f5410 2021-11-20T08:53:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f958dffb700 1 -- 172.21.15.102:0/2321241189 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9594013f00 con 0x7f95980f5410 2021-11-20T08:53:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9598060440 con 0x7f95980f5410 2021-11-20T08:53:25.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.520+0000 7f958dffb700 1 -- 172.21.15.102:0/2321241189 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9594014d90 con 0x7f95980f5410 2021-11-20T08:53:25.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.521+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9598109ab0 con 0x7f95980f5410 2021-11-20T08:53:25.528 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.521+0000 7f958dffb700 1 -- 172.21.15.102:0/2321241189 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95940263a0 con 0x7f95980f5410 2021-11-20T08:53:25.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.522+0000 7f958dffb700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f958402ffb0 0x7f9584032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:25.529 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.522+0000 7f958dffb700 1 -- 172.21.15.102:0/2321241189 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9594040fe0 con 0x7f95980f5410 2021-11-20T08:53:25.530 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.522+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9598045ad0 con 0x7f95980f5410 2021-11-20T08:53:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.522+0000 7f959ce5e700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f958402ffb0 0x7f9584032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.522+0000 7f959ce5e700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f958402ffb0 0x7f9584032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f95880096a0 tx=0x7f9588006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:25.532 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.526+0000 7f958dffb700 1 -- 172.21.15.102:0/2321241189 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9594012920 con 0x7f95980f5410 2021-11-20T08:53:25.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.729+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f959810a1c0 con 0x7f95980f5410 2021-11-20T08:53:25.728 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.729+0000 7f958dffb700 1 -- 172.21.15.102:0/2321241189 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f959402b020 con 0x7f95980f5410 2021-11-20T08:53:25.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:25.729 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","modified":"2021-11-20T08:51:55.545573Z","created":"2021-11-20T08:51:55.545573Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi093","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:3300","nonce":0},{"type":"v1","addr":"172.21.15.93:6789","nonce":0}]},"addr":"172.21.15.93:6789/0","public_addr":"172.21.15.93:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T08:53:25.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f958402ffb0 msgr2=0x7f9584032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:25.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f958402ffb0 0x7f9584032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f95880096a0 tx=0x7f9588006b40).stop 2021-11-20T08:53:25.730 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 msgr2=0x7f9598109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:25.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f9598109400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9594013510 tx=0x7f959400b040).stop 2021-11-20T08:53:25.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 shutdown_connections 2021-11-20T08:53:25.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f958402ffb0 0x7f9584032470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:25.731 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 --2- 172.21.15.102:0/2321241189 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95980f5410 0x7f9598109400 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:25.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 >> 172.21.15.102:0/2321241189 conn(0x7f95980f0b70 msgr2=0x7f95980f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:25.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 shutdown_connections 2021-11-20T08:53:25.732 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:25.731+0000 7f959f8c3700 1 -- 172.21.15.102:0/2321241189 wait complete. 2021-11-20T08:53:25.732 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2021-11-20T08:53:26.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:25 smithi093 conmon[33522]: cluster 2021-11-20T08:53:25.487391+0000 mgr.smithi093.qdikkm (mgr.14162) 23 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:27.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T08:53:27.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T08:53:27.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:26 smithi093 conmon[33522]: audit 2021-11-20T08:53:25.730135+0000 mon.smithi093 (mon.0) 132 : audit 2021-11-20T08:53:27.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:26 smithi093 conmon[33522]: [DBG] from='client.? 172.21.15.102:0/2321241189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:28.088 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:27 smithi093 conmon[33522]: cluster 2021-11-20T08:53:27.487766+0000 mgr.smithi093.qdikkm (mgr.14162) 24 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:28.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.576+0000 7fbbcdb29700 1 -- 172.21.15.102:0/2841198831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 msgr2=0x7fbbc80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:28.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.576+0000 7fbbcdb29700 1 --2- 172.21.15.102:0/2841198831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc80f66c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fbbbc009fb0 tx=0x7fbbbc009b30).stop 2021-11-20T08:53:28.577 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.576+0000 7fbbcdb29700 1 -- 172.21.15.102:0/2841198831 shutdown_connections 2021-11-20T08:53:28.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.576+0000 7fbbcdb29700 1 --2- 172.21.15.102:0/2841198831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc80f66c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:28.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.576+0000 7fbbcdb29700 1 -- 172.21.15.102:0/2841198831 >> 172.21.15.102:0/2841198831 conn(0x7fbbc80f1ac0 msgr2=0x7fbbc80f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:28.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.576+0000 7fbbcdb29700 1 -- 172.21.15.102:0/2841198831 shutdown_connections 2021-11-20T08:53:28.578 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.576+0000 7fbbcdb29700 1 -- 172.21.15.102:0/2841198831 wait complete. 2021-11-20T08:53:28.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.577+0000 7fbbcdb29700 1 Processor -- start 2021-11-20T08:53:28.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.577+0000 7fbbcdb29700 1 -- start start 2021-11-20T08:53:28.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.577+0000 7fbbcdb29700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc8106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:28.579 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.577+0000 7fbbcdb29700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81069f0 con 0x7fbbc80f62a0 2021-11-20T08:53:28.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.577+0000 7fbbccb27700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc8106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:28.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.577+0000 7fbbccb27700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc8106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54932/0 (socket says 172.21.15.102:54932) 2021-11-20T08:53:28.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.577+0000 7fbbccb27700 1 -- 172.21.15.102:0/108754794 learned_addr learned my addr 172.21.15.102:0/108754794 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:28.580 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.578+0000 7fbbccb27700 1 -- 172.21.15.102:0/108754794 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc005040 con 0x7fbbc80f62a0 2021-11-20T08:53:28.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.578+0000 7fbbccb27700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc8106500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fbbbc013510 tx=0x7fbbbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:28.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.578+0000 7fbbc57fa700 1 -- 172.21.15.102:0/108754794 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbbc009320 con 0x7fbbc80f62a0 2021-11-20T08:53:28.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.578+0000 7fbbc57fa700 1 -- 172.21.15.102:0/108754794 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbbc013f00 con 0x7fbbc80f62a0 2021-11-20T08:53:28.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.578+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc8106bf0 con 0x7fbbc80f62a0 2021-11-20T08:53:28.581 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.578+0000 7fbbc57fa700 1 -- 172.21.15.102:0/108754794 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbbc014dc0 con 0x7fbbc80f62a0 2021-11-20T08:53:28.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.578+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc8107080 con 0x7fbbc80f62a0 2021-11-20T08:53:28.582 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.579+0000 7fbbc57fa700 1 -- 172.21.15.102:0/108754794 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbbc0263a0 con 0x7fbbc80f62a0 2021-11-20T08:53:28.583 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.579+0000 7fbbc57fa700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbbb802ffb0 0x7fbbb8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:28.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.579+0000 7fbbc57fa700 1 -- 172.21.15.102:0/108754794 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbbc041040 con 0x7fbbc80f62a0 2021-11-20T08:53:28.584 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.579+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc8045ad0 con 0x7fbbc80f62a0 2021-11-20T08:53:28.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.579+0000 7fbbc7fff700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbbb802ffb0 0x7fbbb8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:28.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.579+0000 7fbbc7fff700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbbb802ffb0 0x7fbbb8032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbbb40096a0 tx=0x7fbbb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:28.585 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.583+0000 7fbbc57fa700 1 -- 172.21.15.102:0/108754794 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbbc012920 con 0x7fbbc80f62a0 2021-11-20T08:53:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.786+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbc8107d30 con 0x7fbbc80f62a0 2021-11-20T08:53:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.786+0000 7fbbc57fa700 1 -- 172.21.15.102:0/108754794 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbbbc02b020 con 0x7fbbc80f62a0 2021-11-20T08:53:28.785 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:28.786 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","modified":"2021-11-20T08:51:55.545573Z","created":"2021-11-20T08:51:55.545573Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi093","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:3300","nonce":0},{"type":"v1","addr":"172.21.15.93:6789","nonce":0}]},"addr":"172.21.15.93:6789/0","public_addr":"172.21.15.93:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T08:53:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbbb802ffb0 msgr2=0x7fbbb8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbbb802ffb0 0x7fbbb8032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbbb40096a0 tx=0x7fbbb4006b40).stop 2021-11-20T08:53:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 msgr2=0x7fbbc8106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc8106500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fbbbc013510 tx=0x7fbbbc00b040).stop 2021-11-20T08:53:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 shutdown_connections 2021-11-20T08:53:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbbb802ffb0 0x7fbbb8032470 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 --2- 172.21.15.102:0/108754794 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc80f62a0 0x7fbbc8106500 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 >> 172.21.15.102:0/108754794 conn(0x7fbbc80f1ac0 msgr2=0x7fbbc80f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 shutdown_connections 2021-11-20T08:53:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:28.788+0000 7fbbcdb29700 1 -- 172.21.15.102:0/108754794 wait complete. 2021-11-20T08:53:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2021-11-20T08:53:29.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:28 smithi093 conmon[33522]: audit 2021-11-20T08:53:28.786976+0000 mon.smithi093 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.102:0/108754794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:30.094 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:29 smithi093 conmon[33522]: cluster 2021-11-20T08:53:29.488155+0000 mgr.smithi093.qdikkm (mgr.14162) 25 : cluster [DBG] 2021-11-20T08:53:30.095 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:29 smithi093 conmon[33522]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:30.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T08:53:30.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T08:53:31.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:31 smithi093 conmon[33522]: audit 2021-11-20T08:53:30.346680+0000 mon.smithi093 (mon. 2021-11-20T08:53:31.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:31 smithi093 conmon[33522]: 0) 134 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:31.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:31 smithi093 conmon[33522]: audit 2021-11-20T08:53:30.347204+0000 mon.smithi093 (mon.0) 135 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:53:31.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:31 smithi093 conmon[33522]: audit 2021-11-20T08:53:30.684102+0000 mon.smithi093 (mon.0) 136 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:31.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:31 smithi093 conmon[33522]: audit 2021-11-20T08:53:31.326608+0000 mon.smithi093 (mon.0) 137 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 -- 172.21.15.102:0/186328578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 msgr2=0x7fa5900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 --2- 172.21.15.102:0/186328578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa5900f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fa580009fb0 tx=0x7fa580009b30).stop 2021-11-20T08:53:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 -- 172.21.15.102:0/186328578 shutdown_connections 2021-11-20T08:53:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 --2- 172.21.15.102:0/186328578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa5900f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 -- 172.21.15.102:0/186328578 >> 172.21.15.102:0/186328578 conn(0x7fa5900f0b70 msgr2=0x7fa5900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 -- 172.21.15.102:0/186328578 shutdown_connections 2021-11-20T08:53:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 -- 172.21.15.102:0/186328578 wait complete. 2021-11-20T08:53:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.782+0000 7fa595fc0700 1 Processor -- start 2021-11-20T08:53:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa595fc0700 1 -- start start 2021-11-20T08:53:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa595fc0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa59010b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa595fc0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa590107210 con 0x7fa5900f5410 2021-11-20T08:53:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa58f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa59010b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa58f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa59010b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54938/0 (socket says 172.21.15.102:54938) 2021-11-20T08:53:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa58f7fe700 1 -- 172.21.15.102:0/3340184193 learned_addr learned my addr 172.21.15.102:0/3340184193 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa58f7fe700 1 -- 172.21.15.102:0/3340184193 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa580005040 con 0x7fa5900f5410 2021-11-20T08:53:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa58f7fe700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa59010b820 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa580013510 tx=0x7fa58000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.783+0000 7fa577fff700 1 -- 172.21.15.102:0/3340184193 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa580009320 con 0x7fa5900f5410 2021-11-20T08:53:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.784+0000 7fa577fff700 1 -- 172.21.15.102:0/3340184193 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa580013f00 con 0x7fa5900f5410 2021-11-20T08:53:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.784+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa590060220 con 0x7fa5900f5410 2021-11-20T08:53:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.784+0000 7fa577fff700 1 -- 172.21.15.102:0/3340184193 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa580014d90 con 0x7fa5900f5410 2021-11-20T08:53:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.784+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59010bd10 con 0x7fa5900f5410 2021-11-20T08:53:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.784+0000 7fa577fff700 1 -- 172.21.15.102:0/3340184193 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5800263a0 con 0x7fa5900f5410 2021-11-20T08:53:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.784+0000 7fa577fff700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa57802ff60 0x7fa578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.784+0000 7fa577fff700 1 -- 172.21.15.102:0/3340184193 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5800410b0 con 0x7fa5900f5410 2021-11-20T08:53:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.785+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa590045ad0 con 0x7fa5900f5410 2021-11-20T08:53:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.785+0000 7fa58effd700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa57802ff60 0x7fa578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.785+0000 7fa58effd700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa57802ff60 0x7fa578032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa5840096a0 tx=0x7fa584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:31.902 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.789+0000 7fa577fff700 1 -- 172.21.15.102:0/3340184193 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa580012920 con 0x7fa5900f5410 2021-11-20T08:53:31.997 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.999+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa59010c440 con 0x7fa5900f5410 2021-11-20T08:53:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:31.999+0000 7fa577fff700 1 -- 172.21.15.102:0/3340184193 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa58002b020 con 0x7fa5900f5410 2021-11-20T08:53:31.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:31.999 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","modified":"2021-11-20T08:51:55.545573Z","created":"2021-11-20T08:51:55.545573Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi093","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:3300","nonce":0},{"type":"v1","addr":"172.21.15.93:6789","nonce":0}]},"addr":"172.21.15.93:6789/0","public_addr":"172.21.15.93:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T08:53:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa57802ff60 msgr2=0x7fa578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa57802ff60 0x7fa578032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa5840096a0 tx=0x7fa584006b40).stop 2021-11-20T08:53:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 msgr2=0x7fa59010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa59010b820 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa580013510 tx=0x7fa58000b040).stop 2021-11-20T08:53:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 shutdown_connections 2021-11-20T08:53:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa57802ff60 0x7fa578032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 --2- 172.21.15.102:0/3340184193 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5900f5410 0x7fa59010b820 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 >> 172.21.15.102:0/3340184193 conn(0x7fa5900f0b70 msgr2=0x7fa5900f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 shutdown_connections 2021-11-20T08:53:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:32.001+0000 7fa595fc0700 1 -- 172.21.15.102:0/3340184193 wait complete. 2021-11-20T08:53:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2021-11-20T08:53:32.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:32 smithi093 conmon[33522]: cluster 2021-11-20T08:53:31.488372+0000 mgr.smithi093.qdikkm (mgr.14162) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:32.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:32 smithi093 conmon[33522]: audit 2021-11-20T08:53:32.000074+0000 mon.smithi093 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.102:0/3340184193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:33.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T08:53:33.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T08:53:33.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:33 smithi093 conmon[33522]: cluster 2021-11-20T08:53:33.488743+0000 mgr.smithi093.qdikkm (mgr.14162) 27 2021-11-20T08:53:33.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:33 smithi093 conmon[33522]: : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:35.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.325+0000 7f640df78700 1 -- 172.21.15.102:0/3714700915 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 msgr2=0x7f64080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:35.330 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.325+0000 7f640df78700 1 --2- 172.21.15.102:0/3714700915 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f64080f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f6400009960 tx=0x7f6400004d10).stop 2021-11-20T08:53:35.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.325+0000 7f640df78700 1 -- 172.21.15.102:0/3714700915 shutdown_connections 2021-11-20T08:53:35.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.325+0000 7f640df78700 1 --2- 172.21.15.102:0/3714700915 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f64080f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:35.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.325+0000 7f640df78700 1 -- 172.21.15.102:0/3714700915 >> 172.21.15.102:0/3714700915 conn(0x7f64080f0b70 msgr2=0x7f64080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:35.331 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.325+0000 7f640df78700 1 -- 172.21.15.102:0/3714700915 shutdown_connections 2021-11-20T08:53:35.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.325+0000 7f640df78700 1 -- 172.21.15.102:0/3714700915 wait complete. 2021-11-20T08:53:35.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.326+0000 7f640df78700 1 Processor -- start 2021-11-20T08:53:35.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.326+0000 7f640df78700 1 -- start start 2021-11-20T08:53:35.332 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.326+0000 7f640df78700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f6408105640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:35.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.326+0000 7f640df78700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6408105b30 con 0x7f64080f5410 2021-11-20T08:53:35.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.326+0000 7f64077fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f6408105640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:35.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.326+0000 7f64077fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f6408105640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54944/0 (socket says 172.21.15.102:54944) 2021-11-20T08:53:35.333 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.326+0000 7f64077fe700 1 -- 172.21.15.102:0/912897863 learned_addr learned my addr 172.21.15.102:0/912897863 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:35.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.327+0000 7f64077fe700 1 -- 172.21.15.102:0/912897863 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6400009640 con 0x7f64080f5410 2021-11-20T08:53:35.334 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.327+0000 7f64077fe700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f6408105640 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6400004a10 tx=0x7f6400004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:35.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.327+0000 7f63effff700 1 -- 172.21.15.102:0/912897863 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6400018070 con 0x7f64080f5410 2021-11-20T08:53:35.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.327+0000 7f63effff700 1 -- 172.21.15.102:0/912897863 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6400007e70 con 0x7f64080f5410 2021-11-20T08:53:35.335 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.327+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6408105d30 con 0x7f64080f5410 2021-11-20T08:53:35.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.327+0000 7f63effff700 1 -- 172.21.15.102:0/912897863 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f640001d420 con 0x7f64080f5410 2021-11-20T08:53:35.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.327+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64081061c0 con 0x7f64080f5410 2021-11-20T08:53:35.336 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.328+0000 7f63effff700 1 -- 172.21.15.102:0/912897863 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f640001d580 con 0x7f64080f5410 2021-11-20T08:53:35.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.328+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6408045ad0 con 0x7f64080f5410 2021-11-20T08:53:35.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.328+0000 7f63effff700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f63f002ff60 0x7f63f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:35.337 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.328+0000 7f63effff700 1 -- 172.21.15.102:0/912897863 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64000419a0 con 0x7f64080f5410 2021-11-20T08:53:35.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.328+0000 7f6406ffd700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f63f002ff60 0x7f63f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:35.338 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.328+0000 7f6406ffd700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f63f002ff60 0x7f63f0032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f63fc0096a0 tx=0x7f63fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:35.339 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.332+0000 7f63effff700 1 -- 172.21.15.102:0/912897863 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6400013630 con 0x7f64080f5410 2021-11-20T08:53:35.559 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.554+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64080fae60 con 0x7f64080f5410 2021-11-20T08:53:35.560 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.555+0000 7f63effff700 1 -- 172.21.15.102:0/912897863 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f640001b7e0 con 0x7f64080f5410 2021-11-20T08:53:35.560 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:35.562 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","modified":"2021-11-20T08:51:55.545573Z","created":"2021-11-20T08:51:55.545573Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi093","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:3300","nonce":0},{"type":"v1","addr":"172.21.15.93:6789","nonce":0}]},"addr":"172.21.15.93:6789/0","public_addr":"172.21.15.93:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T08:53:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.556+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f63f002ff60 msgr2=0x7f63f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.556+0000 7f640df78700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f63f002ff60 0x7f63f0032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f63fc0096a0 tx=0x7f63fc006b40).stop 2021-11-20T08:53:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.556+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 msgr2=0x7f6408105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.556+0000 7f640df78700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f6408105640 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6400004a10 tx=0x7f6400004d10).stop 2021-11-20T08:53:35.563 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.557+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 shutdown_connections 2021-11-20T08:53:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.557+0000 7f640df78700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f63f002ff60 0x7f63f0032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.557+0000 7f640df78700 1 --2- 172.21.15.102:0/912897863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f64080f5410 0x7f6408105640 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.557+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 >> 172.21.15.102:0/912897863 conn(0x7f64080f0b70 msgr2=0x7f64080f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:35.564 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.557+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 shutdown_connections 2021-11-20T08:53:35.565 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:35.557+0000 7f640df78700 1 -- 172.21.15.102:0/912897863 wait complete. 2021-11-20T08:53:35.565 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2021-11-20T08:53:35.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: audit 2021-11-20T08:53:34.416118+0000 mon.smithi093 (mon.0) 139 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:35.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: audit 2021-11-20T08:53:34.508813+0000 mon.smithi093 (mon.0) 140 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:35.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: audit 2021-11-20T08:53:35.040380+0000 mon.smithi093 (mon.0) 141 : audit [INF] 2021-11-20T08:53:35.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:35.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: audit 2021-11-20T08:53:35. 2021-11-20T08:53:35.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: 045706+0000 mon.smithi093 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:35.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:53:35.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: -20T08:53:35.046739+0000 2021-11-20T08:53:35.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: mon.smithi093 (mon.0) 143 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T08:53:35.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: 2021-11-20T08:53:35.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: audit 2021-11-20T08:53:35.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: 2021-11-20T08:53:35.051197+0000 mon.smithi093 2021-11-20T08:53:35.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: (mon.0) 144 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T08:53:35.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:53:35.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: -20T08:53:35.051891+0000 2021-11-20T08:53:35.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: mon.smithi093 (mon.0) 145 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:53:35.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:35 smithi093 conmon[33522]: 2021-11-20T08:53:36.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:36 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:35.052475+0000 mgr.smithi093.qdikkm (mgr.14162) 28 : cephadm [INF] 2021-11-20T08:53:36.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:36 smithi093 conmon[33522]: Deploying daemon crash.smithi102 on smithi102 2021-11-20T08:53:36.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:36 smithi093 conmon[33522]: cluster 2021-11-20T08:53:35.489018+0000 mgr.smithi093.qdikkm (mgr.14162) 29 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:36.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:36 smithi093 conmon[33522]: audit 2021-11-20T08:53:35. 2021-11-20T08:53:36.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:36 smithi093 conmon[33522]: 555335+0000 mon.smithi093 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.102:0/912897863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:36.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T08:53:36.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T08:53:37.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:37 smithi093 conmon[33522]: cluster 2021-11-20T08:53:37.489381+0000 mgr.smithi093.qdikkm (mgr.14162) 30 : cluster [DBG] 2021-11-20T08:53:37.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:37 smithi093 conmon[33522]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:39.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: audit 2021-11-20T08:53:38.947543+0000 mon.smithi093 (mon.0) 147 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:39.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: audit 2021-11-20T08:53:38.948979 2021-11-20T08:53:39.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 148 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.ievjft", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T08:53:39.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: audit 2021-11-20T08:53:38.953652+0000 mon.smithi093 (mon.0) 149 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.ievjft", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-11-20T08:53:39.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: audit 2021-11-20T08:53:38.954561 2021-11-20T08:53:39.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 150 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T08:53:39.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: audit 2021-11-20T08:53:39.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: 2021-11-20T08:53:38.955545+0000 mon.smithi093 (mon.0 2021-11-20T08:53:39.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: ) 151 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:53:39.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: cephadm 2021-11-20 2021-11-20T08:53:39.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: T08:53:38.956464+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:53:39.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: 14162) 31 : cephadm [INF] Deploying daemon mgr.smithi102.ievjft on smithi102 2021-11-20T08:53:39.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: cluster 2021-11-20T08:53 2021-11-20T08:53:39.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:39 smithi093 conmon[33522]: :39.489738+0000 mgr.smithi093.qdikkm (mgr.14162) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:42.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:42 smithi093 conmon[33522]: cluster 2021-11-20T08:53:41.490123+0000 mgr.smithi093.qdikkm (mgr.14162) 33 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:43.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:43 smithi093 conmon[33522]: cluster 2021-11-20T08:53:43.490482+0000 mgr.smithi093.qdikkm (mgr.14162) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.124+0000 7f317bfff700 1 -- 172.21.15.102:0/2588760064 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 msgr2=0x7f317c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.124+0000 7f317bfff700 1 --2- 172.21.15.102:0/2588760064 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c0f55c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f316c009fb0 tx=0x7f316c009b30).stop 2021-11-20T08:53:44.350 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.124+0000 7f317bfff700 1 -- 172.21.15.102:0/2588760064 shutdown_connections 2021-11-20T08:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.124+0000 7f317bfff700 1 --2- 172.21.15.102:0/2588760064 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c0f55c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.124+0000 7f317bfff700 1 -- 172.21.15.102:0/2588760064 >> 172.21.15.102:0/2588760064 conn(0x7f317c0f09a0 msgr2=0x7f317c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.124+0000 7f317bfff700 1 -- 172.21.15.102:0/2588760064 shutdown_connections 2021-11-20T08:53:44.351 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.124+0000 7f317bfff700 1 -- 172.21.15.102:0/2588760064 wait complete. 2021-11-20T08:53:44.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.125+0000 7f317bfff700 1 Processor -- start 2021-11-20T08:53:44.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.125+0000 7f317bfff700 1 -- start start 2021-11-20T08:53:44.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.125+0000 7f317bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c1053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:44.352 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.125+0000 7f317bfff700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c1058c0 con 0x7f317c0f51a0 2021-11-20T08:53:44.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.125+0000 7f317affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c1053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:44.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.125+0000 7f317affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c1053d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54950/0 (socket says 172.21.15.102:54950) 2021-11-20T08:53:44.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.125+0000 7f317affd700 1 -- 172.21.15.102:0/3673749470 learned_addr learned my addr 172.21.15.102:0/3673749470 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:44.353 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.126+0000 7f317affd700 1 -- 172.21.15.102:0/3673749470 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c005040 con 0x7f317c0f51a0 2021-11-20T08:53:44.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.126+0000 7f317affd700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c1053d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f316c009830 tx=0x7f316c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:44.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.126+0000 7f3173fff700 1 -- 172.21.15.102:0/3673749470 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f316c009320 con 0x7f317c0f51a0 2021-11-20T08:53:44.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.126+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317c105ac0 con 0x7f317c0f51a0 2021-11-20T08:53:44.354 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.126+0000 7f3173fff700 1 -- 172.21.15.102:0/3673749470 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f316c0136c0 con 0x7f317c0f51a0 2021-11-20T08:53:44.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.126+0000 7f3173fff700 1 -- 172.21.15.102:0/3673749470 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f316c01ee90 con 0x7f317c0f51a0 2021-11-20T08:53:44.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.126+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317c105f50 con 0x7f317c0f51a0 2021-11-20T08:53:44.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.127+0000 7f3173fff700 1 -- 172.21.15.102:0/3673749470 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f316c01e420 con 0x7f317c0f51a0 2021-11-20T08:53:44.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.127+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317c045ad0 con 0x7f317c0f51a0 2021-11-20T08:53:44.355 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.127+0000 7f3173fff700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f316802ff10 0x7f31680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:44.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.127+0000 7f3173fff700 1 -- 172.21.15.102:0/3673749470 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f316c040f10 con 0x7f317c0f51a0 2021-11-20T08:53:44.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.127+0000 7f31727fc700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f316802ff10 0x7f31680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:44.356 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.128+0000 7f31727fc700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f316802ff10 0x7f31680323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f31640096a0 tx=0x7f3164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:44.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.132+0000 7f3173fff700 1 -- 172.21.15.102:0/3673749470 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f316c0095f0 con 0x7f317c0f51a0 2021-11-20T08:53:44.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.338+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f317c106e00 con 0x7f317c0f51a0 2021-11-20T08:53:44.357 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.338+0000 7f3173fff700 1 -- 172.21.15.102:0/3673749470 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f316c02c2d0 con 0x7f317c0f51a0 2021-11-20T08:53:44.358 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:44.358 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","modified":"2021-11-20T08:51:55.545573Z","created":"2021-11-20T08:51:55.545573Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi093","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:3300","nonce":0},{"type":"v1","addr":"172.21.15.93:6789","nonce":0}]},"addr":"172.21.15.93:6789/0","public_addr":"172.21.15.93:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T08:53:44.359 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f316802ff10 msgr2=0x7f31680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:44.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f316802ff10 0x7f31680323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f31640096a0 tx=0x7f3164006b40).stop 2021-11-20T08:53:44.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 msgr2=0x7f317c1053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:44.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c1053d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f316c009830 tx=0x7f316c00b040).stop 2021-11-20T08:53:44.360 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 shutdown_connections 2021-11-20T08:53:44.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f316802ff10 0x7f31680323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:44.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 --2- 172.21.15.102:0/3673749470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f317c0f51a0 0x7f317c1053d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:44.361 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 >> 172.21.15.102:0/3673749470 conn(0x7f317c0f09a0 msgr2=0x7f317c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:44.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 shutdown_connections 2021-11-20T08:53:44.362 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:44.340+0000 7f317bfff700 1 -- 172.21.15.102:0/3673749470 wait complete. 2021-11-20T08:53:44.362 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2021-11-20T08:53:44.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:44 smithi093 conmon[33522]: audit 2021-11-20T08:53:44.338944+0000 mon.smithi093 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.102:0/3673749470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:45.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:45 smithi093 conmon[33522]: cluster 2021-11-20T08:53:45.490884+0000 mgr.smithi093.qdikkm 2021-11-20T08:53:45.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:45 smithi093 conmon[33522]: (mgr.14162) 35 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:47.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:47 smithi093 conmon[33522]: cluster 2021-11-20T08:53:47.491360+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:53:47.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:47 smithi093 conmon[33522]: 36 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:48.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T08:53:48.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T08:53:49.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: audit 2021-11-20T08:53:48.510653+0000 mon.smithi093 (mon.0) 2021-11-20T08:53:49.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: 153 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:49.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: audit 2021-11-20T08:53:48.511999+0000 mon.smithi093 (mon.0) 154 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T08:53:49.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: audit 2021-11-20T08:53 2021-11-20T08:53:49.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: :48.512867+0000 mon.smithi093 (mon.0) 155 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:53:49.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: cephadm 2021- 2021-11-20T08:53:49.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: 11-20T08:53:48.513596+0000 mgr.smithi093.qdikkm (mgr.14162) 37 : cephadm [INF] Deploying daemon mon.smithi102 on smithi102 2021-11-20T08:53:49.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:53:49.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: 20T08:53:49.491793+0000 mgr.smithi093.qdikkm (mgr.14162) 38 : cluster 2021-11-20T08:53:49.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:49 smithi093 conmon[33522]: [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.005+0000 7f32bf0fa700 1 -- 172.21.15.102:0/1955452460 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 msgr2=0x7f32b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:50.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.005+0000 7f32bf0fa700 1 --2- 172.21.15.102:0/1955452460 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b80f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f32ac009fb0 tx=0x7f32ac009b30).stop 2021-11-20T08:53:50.009 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.005+0000 7f32bf0fa700 1 -- 172.21.15.102:0/1955452460 shutdown_connections 2021-11-20T08:53:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.005+0000 7f32bf0fa700 1 --2- 172.21.15.102:0/1955452460 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b80f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.005+0000 7f32bf0fa700 1 -- 172.21.15.102:0/1955452460 >> 172.21.15.102:0/1955452460 conn(0x7f32b80f1c70 msgr2=0x7f32b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.005+0000 7f32bf0fa700 1 -- 172.21.15.102:0/1955452460 shutdown_connections 2021-11-20T08:53:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.005+0000 7f32bf0fa700 1 -- 172.21.15.102:0/1955452460 wait complete. 2021-11-20T08:53:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.006+0000 7f32bf0fa700 1 Processor -- start 2021-11-20T08:53:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.006+0000 7f32bf0fa700 1 -- start start 2021-11-20T08:53:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.006+0000 7f32bf0fa700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.006+0000 7f32bf0fa700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b8106c20 con 0x7f32b80f6510 2021-11-20T08:53:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.007+0000 7f32bce96700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.007+0000 7f32bce96700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.102:54960/0 (socket says 172.21.15.102:54960) 2021-11-20T08:53:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.007+0000 7f32bce96700 1 -- 172.21.15.102:0/3359007340 learned_addr learned my addr 172.21.15.102:0/3359007340 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.007+0000 7f32bce96700 1 -- 172.21.15.102:0/3359007340 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac005040 con 0x7f32b80f6510 2021-11-20T08:53:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.007+0000 7f32bce96700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b8106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f32ac013510 tx=0x7f32ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.007+0000 7f32b57fa700 1 -- 172.21.15.102:0/3359007340 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32ac009320 con 0x7f32b80f6510 2021-11-20T08:53:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.008+0000 7f32b57fa700 1 -- 172.21.15.102:0/3359007340 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32ac013f00 con 0x7f32b80f6510 2021-11-20T08:53:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.008+0000 7f32b57fa700 1 -- 172.21.15.102:0/3359007340 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32ac014d90 con 0x7f32b80f6510 2021-11-20T08:53:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.008+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b8106e20 con 0x7f32b80f6510 2021-11-20T08:53:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.008+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b81072b0 con 0x7f32b80f6510 2021-11-20T08:53:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.008+0000 7f32b57fa700 1 -- 172.21.15.102:0/3359007340 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32ac0263a0 con 0x7f32b80f6510 2021-11-20T08:53:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.009+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b8045ad0 con 0x7f32b80f6510 2021-11-20T08:53:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.009+0000 7f32b57fa700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32a802ff60 0x7f32a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.009+0000 7f32b57fa700 1 -- 172.21.15.102:0/3359007340 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32ac0410d0 con 0x7f32b80f6510 2021-11-20T08:53:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.009+0000 7f32b7fff700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32a802ff60 0x7f32a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:50.016 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.009+0000 7f32b7fff700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32a802ff60 0x7f32a8032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f32a40096a0 tx=0x7f32a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:50.017 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.013+0000 7f32b57fa700 1 -- 172.21.15.102:0/3359007340 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32ac0126f0 con 0x7f32b80f6510 2021-11-20T08:53:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.244+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32b80fbf60 con 0x7f32b80f6510 2021-11-20T08:53:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.245+0000 7f32b57fa700 1 -- 172.21.15.102:0/3359007340 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32ac012330 con 0x7f32b80f6510 2021-11-20T08:53:50.245 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:50.245 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","modified":"2021-11-20T08:51:55.545573Z","created":"2021-11-20T08:51:55.545573Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi093","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:3300","nonce":0},{"type":"v1","addr":"172.21.15.93:6789","nonce":0}]},"addr":"172.21.15.93:6789/0","public_addr":"172.21.15.93:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T08:53:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.246+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32a802ff60 msgr2=0x7f32a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.246+0000 7f32bf0fa700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32a802ff60 0x7f32a8032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f32a40096a0 tx=0x7f32a4006b40).stop 2021-11-20T08:53:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.246+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 msgr2=0x7f32b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.246+0000 7f32bf0fa700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b8106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f32ac013510 tx=0x7f32ac00b040).stop 2021-11-20T08:53:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.247+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 shutdown_connections 2021-11-20T08:53:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.247+0000 7f32bf0fa700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32a802ff60 0x7f32a8032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.247+0000 7f32bf0fa700 1 --2- 172.21.15.102:0/3359007340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32b80f6510 0x7f32b8106730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.247+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 >> 172.21.15.102:0/3359007340 conn(0x7f32b80f1c70 msgr2=0x7f32b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.247+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 shutdown_connections 2021-11-20T08:53:50.249 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:50.247+0000 7f32bf0fa700 1 -- 172.21.15.102:0/3359007340 wait complete. 2021-11-20T08:53:50.250 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2021-11-20T08:53:50.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:50 smithi093 conmon[33522]: audit 2021-11-20T08:53:50.245390+0000 mon.smithi093 (mon.0) 156 : audit [DBG] 2021-11-20T08:53:50.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:50 smithi093 conmon[33522]: from='client.? 172.21.15.102:0/3359007340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:51.340 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 systemd[1]: Starting Ceph mon.smithi102 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T08:53:51.620 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 podman[36431]: 2021-11-20T08:53:51.621 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.536+0000 7f918a5bd700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T08:53:51.621 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.536+0000 7f918a5bd700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-20T08:53:51.621 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.536+0000 7f918a5bd700 0 pidfile_write: ignore empty --pid-file 2021-11-20T08:53:51.621 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.540+0000 7f918a5bd700 0 load: jerasure load: lrc load: isa 2021-11-20T08:53:51.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T08:53:51.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T08:53:51.622 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Compile date May 13 2021 2021-11-20T08:53:51.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: DB SUMMARY 2021-11-20T08:53:51.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.623 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T08:53:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T08:53:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.625 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-11-20T08:53:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi102/store.db dir, Total Num: 0, files: 2021-11-20T08:53:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi102/store.db: 000003.log size: 511 ; 2021-11-20T08:53:51.626 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T08:53:51.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T08:53:51.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T08:53:51.627 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.env: 0x564e87c641c0 2021-11-20T08:53:51.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.fs: Posix File System 2021-11-20T08:53:51.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.info_log: 0x564e896e1960 2021-11-20T08:53:51.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T08:53:51.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.statistics: (nil) 2021-11-20T08:53:51.628 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.use_fsync: 0 2021-11-20T08:53:51.630 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T08:53:51.630 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T08:53:51.631 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T08:53:51.631 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T08:53:51.631 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T08:53:51.631 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T08:53:51.631 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T08:53:51.632 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T08:53:51.632 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T08:53:51.632 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T08:53:51.632 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T08:53:51.633 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.db_log_dir: 2021-11-20T08:53:51.633 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi102/store.db 2021-11-20T08:53:51.633 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T08:53:51.633 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T08:53:51.634 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T08:53:51.634 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T08:53:51.634 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T08:53:51.634 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T08:53:51.634 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T08:53:51.635 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T08:53:51.635 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T08:53:51.635 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T08:53:51.635 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.write_buffer_manager: 0x564e8a356030 2021-11-20T08:53:51.635 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T08:53:51.636 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T08:53:51.636 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 bash[36351]: fba614458a87daf7ea8e4117ff598500031313c16ace9b8ec3c962cc15a66e9d 2021-11-20T08:53:51.636 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T08:53:51.636 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T08:53:51.637 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T08:53:51.637 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T08:53:51.637 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T08:53:51.637 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T08:53:51.637 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T08:53:51.638 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.unordered_write: 0 2021-11-20T08:53:51.638 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T08:53:51.638 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.639 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T08:53:51.640 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T08:53:51.640 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T08:53:51.641 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.row_cache: None 2021-11-20T08:53:51.642 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.wal_filter: None 2021-11-20T08:53:51.642 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T08:53:51.643 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T08:53:51.643 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T08:53:51.643 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T08:53:51.643 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T08:53:51.645 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T08:53:51.645 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T08:53:51.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T08:53:51.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T08:53:51.647 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T08:53:51.647 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T08:53:51.648 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T08:53:51.648 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T08:53:51.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T08:53:51.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T08:53:51.649 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T08:53:51.659 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T08:53:51.660 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T08:53:51.660 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T08:53:51.661 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.661 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T08:53:51.662 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T08:53:51.662 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_open_files: -1 2021-11-20T08:53:51.677 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T08:53:51.677 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T08:53:51.678 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T08:53:51.678 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T08:53:51.678 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Compression algorithms supported: 2021-11-20T08:53:51.679 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T08:53:51.679 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kZSTD supported: 0 2021-11-20T08:53:51.687 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T08:53:51.687 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T08:53:51.687 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T08:53:51.687 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T08:53:51.688 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T08:53:51.688 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T08:53:51.688 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T08:53:51.688 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000001 2021-11-20T08:53:51.688 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.689 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T08:53:51.689 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.689 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T08:53:51.689 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.merge_operator: 2021-11-20T08:53:51.690 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_filter: None 2021-11-20T08:53:51.690 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T08:53:51.690 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T08:53:51.696 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T08:53:51.696 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564e895f8128) 2021-11-20T08:53:51.697 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: cache_index_and_filter_blocks: 1 2021-11-20T08:53:51.709 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T08:53:51.709 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T08:53:51.709 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: pin_top_level_index_and_filter: 1 2021-11-20T08:53:51.710 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: index_type: 0 2021-11-20T08:53:51.710 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: data_block_index_type: 0 2021-11-20T08:53:51.710 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: index_shortening: 1 2021-11-20T08:53:51.711 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T08:53:51.712 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: hash_index_allow_collision: 1 2021-11-20T08:53:51.712 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: checksum: 1 2021-11-20T08:53:51.713 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: no_block_cache: 0 2021-11-20T08:53:51.713 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_cache: 0x564e8962ef10 2021-11-20T08:53:51.713 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_cache_name: BinnedLRUCache 2021-11-20T08:53:51.714 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_cache_options: 2021-11-20T08:53:51.714 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: capacity : 536870912 2021-11-20T08:53:51.714 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: num_shard_bits : 4 2021-11-20T08:53:51.714 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: strict_capacity_limit : 0 2021-11-20T08:53:51.714 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: high_pri_pool_ratio: 0.000 2021-11-20T08:53:51.715 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_cache_compressed: (nil) 2021-11-20T08:53:51.715 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: persistent_cache: (nil) 2021-11-20T08:53:51.717 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_size: 4096 2021-11-20T08:53:51.717 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_size_deviation: 10 2021-11-20T08:53:51.717 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_restart_interval: 16 2021-11-20T08:53:51.717 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: index_block_restart_interval: 1 2021-11-20T08:53:51.718 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: metadata_block_size: 4096 2021-11-20T08:53:51.718 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: partition_filters: 0 2021-11-20T08:53:51.719 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: use_delta_encoding: 1 2021-11-20T08:53:51.719 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T08:53:51.719 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: whole_key_filtering: 1 2021-11-20T08:53:51.720 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: verify_compression: 0 2021-11-20T08:53:51.720 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: read_amp_bytes_per_bit: 0 2021-11-20T08:53:51.720 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: format_version: 2 2021-11-20T08:53:51.720 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: enable_index_compression: 1 2021-11-20T08:53:51.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: block_align: 0 2021-11-20T08:53:51.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T08:53:51.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T08:53:51.721 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compression: NoCompression 2021-11-20T08:53:51.722 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T08:53:51.722 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T08:53:51.722 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T08:53:51.723 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.num_levels: 7 2021-11-20T08:53:51.723 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T08:53:51.723 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T08:53:51.723 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T08:53:51.724 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T08:53:51.724 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T08:53:51.724 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T08:53:51.724 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T08:53:51.725 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T08:53:51.725 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T08:53:51.725 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T08:53:51.725 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T08:53:51.726 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T08:53:51.726 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T08:53:51.726 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T08:53:51.726 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T08:53:51.727 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T08:53:51.727 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T08:53:51.727 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T08:53:51.727 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T08:53:51.727 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T08:53:51.728 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T08:53:51.728 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T08:53:51.728 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T08:53:51.728 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T08:53:51.728 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T08:53:51.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T08:53:51.730 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T08:53:51.731 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T08:53:51.731 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T08:53:51.731 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T08:53:51.732 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T08:53:51.732 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T08:53:51.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T08:53:51.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T08:53:51.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T08:53:51.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T08:53:51.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T08:53:51.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T08:53:51.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T08:53:51.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T08:53:51.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T08:53:51.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T08:53:51.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T08:53:51.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T08:53:51.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T08:53:51.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T08:53:51.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T08:53:51.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T08:53:51.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T08:53:51.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T08:53:51.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T08:53:51.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T08:53:51.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T08:53:51.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T08:53:51.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T08:53:51.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T08:53:51.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T08:53:51.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T08:53:51.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T08:53:51.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.ttl: 2592000 2021-11-20T08:53:51.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.541+0000 7f918a5bd700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T08:53:51.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.542+0000 7f918a5bd700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T08:53:51.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.542+0000 7f918a5bd700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T08:53:51.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.542+0000 7f918a5bd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637398431544035, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-11-20T08:53:51.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.542+0000 7f918a5bd700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-11-20T08:53:51.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.542+0000 7f918a5bd700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T08:53:51.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.543+0000 7f918a5bd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637398431544510, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637398431, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T08:53:51.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.543+0000 7f918a5bd700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-11-20T08:53:51.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.543+0000 7f918a5bd700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637398431544937, "job": 1, "event": "recovery_finished"} 2021-11-20T08:53:51.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.544+0000 7f918a5bd700 4 rocksdb: DB pointer 0x564e896f3800 2021-11-20T08:53:51.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.544+0000 7f91713d5700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T08:53:51.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.544+0000 7f91713d5700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T08:53:51.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ** DB Stats ** 2021-11-20T08:53:51.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:53:51.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T08:53:51.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T08:53:51.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T08:53:51.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T08:53:51.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T08:53:51.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T08:53:51.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ** Compaction Stats [default] ** 2021-11-20T08:53:51.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:53:51.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:53:51.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T08:53:51.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T08:53:51.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T08:53:51.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ** Compaction Stats [default] ** 2021-11-20T08:53:51.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:53:51.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:53:51.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T08:53:51.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:53:51.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T08:53:51.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T08:53:51.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T08:53:51.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T08:53:51.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T08:53:51.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:53:51.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:53:51.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T08:53:51.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T08:53:51.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ** Compaction Stats [default] ** 2021-11-20T08:53:51.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:53:51.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:53:51.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T08:53:51.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T08:53:51.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T08:53:51.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ** Compaction Stats [default] ** 2021-11-20T08:53:51.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T08:53:51.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T08:53:51.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T08:53:51.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T08:53:51.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T08:53:51.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T08:53:51.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T08:53:51.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T08:53:51.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T08:53:51.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:53:51.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T08:53:51.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T08:53:51.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T08:53:51.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.544+0000 7f918a5bd700 0 mon.smithi102 does not exist in monmap, will attempt to join an existing cluster 2021-11-20T08:53:51.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.544+0000 7f918a5bd700 0 using public_addr v2:172.21.15.102:0/0 -> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] 2021-11-20T08:53:51.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.545+0000 7f918a5bd700 0 starting mon.smithi102 rank -1 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi102 fsid 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:53:51.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.546+0000 7f918a5bd700 1 mon.smithi102@-1(???) e0 preinit fsid 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:53:51.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 0 mon.smithi102@-1(synchronizing).mds e1 new map 2021-11-20T08:53:51.765 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 0 mon.smithi102@-1(synchronizing).mds e1 print_map 2021-11-20T08:53:51.765 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: e1 2021-11-20T08:53:51.765 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T08:53:51.765 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T08:53:51.766 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: legacy client fscid: -1 2021-11-20T08:53:51.766 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.766 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: No filesystems configured 2021-11-20T08:53:51.766 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.767 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-11-20T08:53:51.767 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-11-20T08:53:51.767 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-11-20T08:53:51.767 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-11-20T08:53:51.768 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-11-20T08:53:51.768 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:51.768 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-11-20T08:53:51.768 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.570+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2021-11-20T08:53:51.768 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.572+0000 7f9173bda700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T08:53:51.769 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.572+0000 7f9173bda700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T08:53:51.769 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.572+0000 7f9173bda700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T08:53:51.769 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.572+0000 7f9173bda700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T08:53:51.769 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: audit 2021-11-20T08:53:51.770 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: 2021-11-20T08:53:50.245390+0000 2021-11-20T08:53:51.770 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: mon.smithi093 (mon.0) 156 : audit [DBG] 2021-11-20T08:53:51.770 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: from='client.? 172.21.15.102:0/3359007340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:51.770 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 conmon[36472]: debug 2021-11-20T08:53:51.572+0000 7f9173bda700 1 mon.smithi102@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2021-11-20T08:53:51.770 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:51 smithi102 systemd[1]: Started Ceph mon.smithi102 for 11458b34-49df-11ec-8c2c-001a4aab830c. 2021-11-20T08:53:51.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T08:53:51.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T08:53:56.907 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.905+0000 7f7c09d91700 1 -- 172.21.15.102:0/4006795990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 msgr2=0x7f7bf40056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.905+0000 7f7c09d91700 1 --2- 172.21.15.102:0/4006795990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 0x7f7bf40056e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7c040f70f0 tx=0x7f7bf800b040).stop 2021-11-20T08:53:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.905+0000 7f7c09d91700 1 -- 172.21.15.102:0/4006795990 shutdown_connections 2021-11-20T08:53:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.905+0000 7f7c09d91700 1 --2- 172.21.15.102:0/4006795990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 0x7f7bf40056e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:56.908 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.905+0000 7f7c09d91700 1 -- 172.21.15.102:0/4006795990 >> 172.21.15.102:0/4006795990 conn(0x7f7c040f1aa0 msgr2=0x7f7c040f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:56.909 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 -- 172.21.15.102:0/4006795990 shutdown_connections 2021-11-20T08:53:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 -- 172.21.15.102:0/4006795990 wait complete. 2021-11-20T08:53:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 Processor -- start 2021-11-20T08:53:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 -- start start 2021-11-20T08:53:56.910 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 0x7f7c0406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:56.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 0x7f7c0406a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:56.911 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0406a820 con 0x7f7bf4003250 2021-11-20T08:53:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.906+0000 7f7c09d91700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0406a960 con 0x7f7c04069e20 2021-11-20T08:53:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c03fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 0x7f7c0406a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c03fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 0x7f7c0406a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53674/0 (socket says 172.21.15.102:53674) 2021-11-20T08:53:56.913 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c03fff700 1 -- 172.21.15.102:0/2085841947 learned_addr learned my addr 172.21.15.102:0/2085841947 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:53:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c03fff700 1 -- 172.21.15.102:0/2085841947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 msgr2=0x7f7c0406a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2021-11-20T08:53:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c03fff700 1 -- 172.21.15.102:0/2085841947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 msgr2=0x7f7c0406a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-11-20T08:53:56.914 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c03fff700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 0x7f7c0406a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:53:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c03fff700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 0x7f7c0406a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T08:53:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c08d8f700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 0x7f7c0406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c08d8f700 1 -- 172.21.15.102:0/2085841947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 msgr2=0x7f7c0406a2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T08:53:56.915 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c08d8f700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 0x7f7c0406a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c08d8f700 1 -- 172.21.15.102:0/2085841947 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf8005040 con 0x7f7bf4003250 2021-11-20T08:53:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c08d8f700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 0x7f7c0406b9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f7c040f70f0 tx=0x7f7bf8002040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c017fa700 1 -- 172.21.15.102:0/2085841947 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf8004450 con 0x7f7bf4003250 2021-11-20T08:53:56.916 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0406abb0 con 0x7f7bf4003250 2021-11-20T08:53:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c017fa700 1 -- 172.21.15.102:0/2085841947 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bf8017410 con 0x7f7bf4003250 2021-11-20T08:53:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.907+0000 7f7c017fa700 1 -- 172.21.15.102:0/2085841947 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf8021df0 con 0x7f7bf4003250 2021-11-20T08:53:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.908+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0410e8d0 con 0x7f7bf4003250 2021-11-20T08:53:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.908+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c04045ad0 con 0x7f7bf4003250 2021-11-20T08:53:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.909+0000 7f7c017fa700 1 -- 172.21.15.102:0/2085841947 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7bf8031070 con 0x7f7bf4003250 2021-11-20T08:53:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.909+0000 7f7c017fa700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7be805c3a0 0x7f7be805e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.909+0000 7f7c017fa700 1 -- 172.21.15.102:0/2085841947 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bf8077920 con 0x7f7bf4003250 2021-11-20T08:53:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.909+0000 7f7c03fff700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7be805c3a0 0x7f7be805e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.910+0000 7f7c03fff700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7be805c3a0 0x7f7be805e860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7bf0006cf0 tx=0x7f7bf0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:56.914+0000 7f7c017fa700 1 -- 172.21.15.102:0/2085841947 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bf804b5f0 con 0x7f7bf4003250 2021-11-20T08:53:56.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53:51.580493+0000 mon.smithi093 ( 2021-11-20T08:53:56.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: mon.0) 158 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi093"}]: dispatch 2021-11-20T08:53:56.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11-20T08:53:51.580691+0000 mon.smithi093 (mon.0) 159 : cluster [INF] mon.smithi093 calling monitor election 2021-11-20T08:53:56.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53:51.593724+0000 mon.smithi093 2021-11-20T08:53:56.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: (mon.0) 160 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:56.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53:51. 2021-11-20T08:53:56.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 619163+0000 mon.smithi093 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:53:56.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:53:56.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 20T08:53:51.621840+0000 mon.smithi093 (mon.0) 162 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:53:56.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53: 2021-11-20T08:53:56.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 52.577860+0000 mon.smithi093 (mon.0) 163 : 2021-11-20T08:53:56.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:56.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11-20T08:53:53.492587 2021-11-20T08:53:56.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 40 : cluster [DBG] 2021-11-20T08:53:56.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:56.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53:53. 2021-11-20T08:53:56.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 578069+0000 mon.smithi093 (mon.0) 164 : audit [DBG] 2021-11-20T08:53:56.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:56.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11-20T08:53:53 2021-11-20T08:53:56.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: .578878+0000 mon.smithi102 (mon.1) 1 : cluster 2021-11-20T08:53:56.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: [INF] mon.smithi102 calling monitor election 2021-11-20T08:53:56.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53 2021-11-20T08:53:56.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: :53.638846+0000 mon.smithi093 (mon.0) 165 2021-11-20T08:53:56.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: : audit [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.ievjft/crt"}]: dispatch 2021-11-20T08:53:56.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:53:56.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 08:53:54.578082+0000 mon.smithi093 (mon. 2021-11-20T08:53:56.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 0) 166 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:56.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:53:56.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 11-20T08:53:55.492973+0000 2021-11-20T08:53:56.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 41 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:56.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 2021-11-20T08:53:56.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53:56.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 2021-11-20T08:53:55.578071+0000 mon.smithi093 2021-11-20T08:53:56.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: (mon.0) 167 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:56.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021 2021-11-20T08:53:56.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: -11-20T08:53:56.578353+0000 2021-11-20T08:53:56.972 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: mon.smithi093 (mon.0) 168 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:56.972 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 2021-11-20T08:53:56.972 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11-20T08:53:56.972 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 2021-11-20T08:53:56.607647+0000 mon.smithi093 2021-11-20T08:53:56.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: (mon.0) 169 : cluster [INF] mon.smithi093 is new leader, mons smithi093,smithi102 in quorum (ranks 0,1) 2021-11-20T08:53:56.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:53:56.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: -11-20T08:53:56.615702+0000 2021-11-20T08:53:56.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: mon.smithi093 (mon.0) 170 : cluster [DBG] monmap e2: 2 mons at {smithi093=[v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0],smithi102=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} 2021-11-20T08:53:56.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:53:56.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 11-20T08:53:56.615770+0000 2021-11-20T08:53:56.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: mon.smithi093 (mon.0) 171 : cluster [DBG] fsmap 2021-11-20T08:53:56.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 2021-11-20T08:53:56.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11-20T08:53:56 2021-11-20T08:53:56.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: .615809+0000 mon.smithi093 (mon.0) 172 : cluster 2021-11-20T08:53:56.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T08:53:56.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:53:56.976 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 53:56.616039+0000 mon.smithi093 (mon.0 2021-11-20T08:53:56.976 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: ) 173 : cluster [DBG] mgrmap e16: smithi093.qdikkm(active, since 65s) 2021-11-20T08:53:56.976 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11-20T08:53:56.617234 2021-11-20T08:53:56.976 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 174 : cluster [DBG] 2021-11-20T08:53:56.977 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: Standby manager daemon smithi102.ievjft started 2021-11-20T08:53:56.977 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53: 2021-11-20T08:53:56.977 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 56.618090+0000 mon.smithi093 (mon.0) 175 : 2021-11-20T08:53:56.977 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-20T08:53:56.977 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08:53:56.619428 2021-11-20T08:53:56.978 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 176 : audit [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.ievjft/key"}]: dispatch 2021-11-20T08:53:56.978 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 2021-11-20T08:53:56.978 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021-11-20T08 2021-11-20T08:53:56.979 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: :53:56.619827+0000 mon.smithi093 (mon.0 2021-11-20T08:53:56.979 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: ) 177 : audit [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-20T08:53:56.979 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:53:56.979 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: 20T08:53:56.623274+0000 mon.smithi093 (mon 2021-11-20T08:53:56.980 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: .0) 178 : cluster [INF] overall HEALTH_OK 2021-11-20T08:53:56.980 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: audit 2021 2021-11-20T08:53:56.980 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: -11-20T08:53:56.628533+0000 2021-11-20T08:53:56.980 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: mon.smithi093 (mon.0) 179 : audit [INF] 2021-11-20T08:53:56.981 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:56 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:57.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53:51.580493+0000 mon.smithi093 (mon.0) 158 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi093"}]: dispatch 2021-11-20T08:53:57.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 2021-11-20T08:53:57.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20T08:53:51.580691+0000 mon.smithi093 (mon.0) 159 : cluster [INF] mon.smithi093 calling monitor election 2021-11-20T08:53:57.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53:51.593724+0000 mon.smithi093 (mon.0) 160 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:57.077 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53:51.619163+0000 mon.smithi093 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:53:57.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53:51.621840+0000 mon.smithi093 (mon.0) 162 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:53:57.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53:52.577860+0000 mon.smithi093 (mon.0) 163 : audit 2021-11-20T08:53:57.078 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:57.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20T08:53:53.492587+0000 mgr.smithi093.qdikkm (mgr.14162) 40 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:57.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 2021-11-20T08:53:57.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:53:57.079 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 20T08:53:53.578069+0000 mon.smithi093 (mon.0) 164 2021-11-20T08:53:57.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:57.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20T08:53 2021-11-20T08:53:57.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :53.578878+0000 mon.smithi102 (mon.1) 1 : 2021-11-20T08:53:57.080 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster [INF] mon.smithi102 calling monitor election 2021-11-20T08:53:57.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53: 2021-11-20T08:53:57.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 53.638846+0000 mon.smithi093 (mon.0) 165 : audit 2021-11-20T08:53:57.081 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.ievjft/crt"}]: dispatch 2021-11-20T08:53:57.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53 2021-11-20T08:53:57.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :54.578082+0000 mon.smithi093 (mon.0) 166 2021-11-20T08:53:57.082 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:57.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20T08 2021-11-20T08:53:57.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :53:55.492973+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:53:57.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 14162) 41 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:57.083 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11 2021-11-20T08:53:57.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: -20T08:53:55.578071+0000 mon.smithi093 2021-11-20T08:53:57.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: (mon.0) 167 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:57.084 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021 2021-11-20T08:53:57.085 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: -11-20T08:53:56.578353+0000 2021-11-20T08:53:57.085 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: mon.smithi093 (mon.0) 168 : audit 2021-11-20T08:53:57.085 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:57.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 2021-11-20T08:53:57.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20T08 2021-11-20T08:53:57.086 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :53:56.607647+0000 mon.smithi093 (mon.0) 2021-11-20T08:53:57.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 169 : cluster [INF] mon.smithi093 is new leader, mons smithi093,smithi102 in quorum (ranks 0,1) 2021-11-20T08:53:57.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:53:57.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: T08:53:56.615702+0000 mon.smithi093 (mon 2021-11-20T08:53:57.087 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: .0) 170 : cluster [DBG] monmap e2: 2 mons at {smithi093=[v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0],smithi102=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} 2021-11-20T08:53:57.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20T08:53:57.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 2021-11-20T08:53 2021-11-20T08:53:57.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :56.615770+0000 mon.smithi093 (mon.0) 171 2021-11-20T08:53:57.088 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: : cluster [DBG] fsmap 2021-11-20T08:53:57.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T08:53:57.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: -20T08:53:56.615809 2021-11-20T08:53:57.089 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 172 : cluster [DBG] 2021-11-20T08:53:57.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: osdmap e5: 0 total, 0 up, 0 in 2021-11-20T08:53:57.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:53:57.090 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: T08:53:56.616039+0000 mon.smithi093 2021-11-20T08:53:57.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: (mon.0) 173 : cluster [DBG] mgrmap e16: smithi093.qdikkm(active, since 65s) 2021-11-20T08:53:57.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021-11-20T08:53:57.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 2021-11-20T08:53 2021-11-20T08:53:57.091 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :56.617234+0000 mon.smithi093 (mon.0) 174 2021-11-20T08:53:57.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: : cluster [DBG] Standby manager daemon smithi102.ievjft started 2021-11-20T08:53:57.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:53:57.092 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 11-20T08:53:56. 2021-11-20T08:53:57.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 618090+0000 mon.smithi093 (mon.0) 175 : audit 2021-11-20T08:53:57.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-20T08:53:57.093 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53 2021-11-20T08:53:57.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :56.619428+0000 mon.smithi093 (mon 2021-11-20T08:53:57.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: .0) 176 : audit [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.ievjft/key"}]: dispatch 2021-11-20T08:53:57.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021-11-20T08:53:57.094 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 2021-11-20T08:53 2021-11-20T08:53:57.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: :56.619827+0000 mon.smithi093 (mon.0) 2021-11-20T08:53:57.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 177 : audit [DBG] from='mgr.? 172.21.15.102:0/1070631401' entity='mgr.smithi102.ievjft' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-20T08:53:57.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: cluster 2021- 2021-11-20T08:53:57.095 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 11-20T08:53:56.623274+0000 mon.smithi093 2021-11-20T08:53:57.096 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: (mon.0) 178 : cluster [INF] overall HEALTH_OK 2021-11-20T08:53:57.096 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit 2021 2021-11-20T08:53:57.096 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: -11-20T08:53: 2021-11-20T08:53:57.097 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: 56.628533+0000 mon.smithi093 (mon.0) 179 : 2021-11-20T08:53:57.097 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:56 smithi102 conmon[36472]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:53:57.141 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.138+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c04027a00 con 0x7f7bf4003250 2021-11-20T08:53:57.142 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.139+0000 7f7c017fa700 1 -- 172.21.15.102:0/2085841947 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+968 (secure 0 0 0) 0x7f7bf8025030 con 0x7f7bf4003250 2021-11-20T08:53:57.142 INFO:teuthology.orchestra.run.smithi102.stdout: 2021-11-20T08:53:57.143 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":2,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","modified":"2021-11-20T08:53:51.577968Z","created":"2021-11-20T08:51:55.545573Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi093","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:3300","nonce":0},{"type":"v1","addr":"172.21.15.93:6789","nonce":0}]},"addr":"172.21.15.93:6789/0","public_addr":"172.21.15.93:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-11-20T08:53:57.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7be805c3a0 msgr2=0x7f7be805e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:57.144 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7be805c3a0 0x7f7be805e860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7bf0006cf0 tx=0x7f7bf0008040).stop 2021-11-20T08:53:57.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 msgr2=0x7f7c0406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:57.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 0x7f7c0406b9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f7c040f70f0 tx=0x7f7bf8002040).stop 2021-11-20T08:53:57.145 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 shutdown_connections 2021-11-20T08:53:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7be805c3a0 0x7f7be805e860 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7bf4003250 0x7f7c0406b9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 --2- 172.21.15.102:0/2085841947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c04069e20 0x7f7c0406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 >> 172.21.15.102:0/2085841947 conn(0x7f7c040f1aa0 msgr2=0x7f7c040fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:57.146 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 shutdown_connections 2021-11-20T08:53:57.147 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:53:57.141+0000 7f7c09d91700 1 -- 172.21.15.102:0/2085841947 wait complete. 2021-11-20T08:53:57.147 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 2 2021-11-20T08:53:57.556 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-11-20T08:53:57.556 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph config generate-minimal-conf 2021-11-20T08:53:57.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:57 smithi102 conmon[36472]: cephadm 2021-11-20T08:53:56.629536+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:53:57.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:57 smithi102 conmon[36472]: mgr.14162) 42 : cephadm [INF] Deploying daemon node-exporter.smithi102 on smithi102 2021-11-20T08:53:57.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:57 smithi102 conmon[36472]: cluster 2021-11-20T08:53:56.677695+0000 mon.smithi093 (mon.0) 180 : cluster [DBG] mgrmap e17: smithi093.qdikkm(active, since 65s), standbys: smithi102.ievjft 2021-11-20T08:53:57.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:57 smithi102 conmon[36472]: audit 2021-11-20T08:53:56.677859+0000 mon.smithi093 (mon.0) 181 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr metadata", "who": "smithi102.ievjft", "id": "smithi102.ievjft"}]: dispatch 2021-11-20T08:53:57.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:57 smithi102 conmon[36472]: audit 2021-11-20T08:53:57.140739+0000 mon.smithi093 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.102:0/2085841947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:57.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:57 smithi102 conmon[36472]: cluster 2021-11-20T08:53:57.493270+0000 mgr.smithi093.qdikkm (mgr.14162) 43 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:57.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:57 smithi102 conmon[36472]: audit 2021-11-20T08:53:57.578758+0000 mon.smithi093 (mon.0) 183 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:57 smithi093 conmon[33522]: cephadm 2021-11-20T08:53:56.629536+0000 2021-11-20T08:53:57.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:57 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 42 : cephadm [INF] Deploying daemon node-exporter.smithi102 on smithi102 2021-11-20T08:53:57.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:57 smithi093 conmon[33522]: cluster 2021-11-20T08:53:56.677695+0000 mon.smithi093 (mon.0) 180 : cluster [DBG] mgrmap e17: smithi093.qdikkm(active, since 65s), standbys: smithi102.ievjft 2021-11-20T08:53:57.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:57 smithi093 conmon[33522]: audit 2021-11-20T08:53:56.677859+0000 mon.smithi093 (mon.0) 181 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr metadata", "who": "smithi102.ievjft", "id": "smithi102.ievjft"}]: dispatch 2021-11-20T08:53:57.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:57 smithi093 conmon[33522]: audit 2021-11-20T08:53:57.140739+0000 mon.smithi093 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.102:0/2085841947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T08:53:57.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:57 smithi093 conmon[33522]: cluster 2021-11-20T08:53:57.493270+0000 mgr.smithi093.qdikkm (mgr.14162) 43 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:53:57.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:57 smithi093 conmon[33522]: audit 2021-11-20T08:53:57.578758+0000 mon.smithi093 (mon.0) 183 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2021-11-20T08:53:59.120 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 -- 172.21.15.93:0/2896679304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 msgr2=0x7f0a840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:59.121 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 --2- 172.21.15.93:0/2896679304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a840f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0a74009960 tx=0x7f0a74004d10).stop 2021-11-20T08:53:59.121 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 -- 172.21.15.93:0/2896679304 shutdown_connections 2021-11-20T08:53:59.121 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 --2- 172.21.15.93:0/2896679304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a840f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:59.121 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 -- 172.21.15.93:0/2896679304 >> 172.21.15.93:0/2896679304 conn(0x7f0a840f1c70 msgr2=0x7f0a840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:59.121 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 -- 172.21.15.93:0/2896679304 shutdown_connections 2021-11-20T08:53:59.122 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 -- 172.21.15.93:0/2896679304 wait complete. 2021-11-20T08:53:59.122 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.119+0000 7f0a89a94700 1 Processor -- start 2021-11-20T08:53:59.122 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a89a94700 1 -- start start 2021-11-20T08:53:59.122 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a89a94700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a8406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:59.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a89a94700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a84069e20 0x7f0a8406a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:59.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a89a94700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8406a820 con 0x7f0a840f6510 2021-11-20T08:53:59.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a89a94700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8406a960 con 0x7f0a84069e20 2021-11-20T08:53:59.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a837fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a8406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:59.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a837fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a8406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42514/0 (socket says 172.21.15.93:42514) 2021-11-20T08:53:59.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a837fe700 1 -- 172.21.15.93:0/3232751761 learned_addr learned my addr 172.21.15.93:0/3232751761 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:53:59.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a837fe700 1 -- 172.21.15.93:0/3232751761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a84069e20 msgr2=0x7f0a8406a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:59.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a82ffd700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a84069e20 0x7f0a8406a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:59.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a837fe700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a84069e20 0x7f0a8406a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:59.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a837fe700 1 -- 172.21.15.93:0/3232751761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74009640 con 0x7f0a840f6510 2021-11-20T08:53:59.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a82ffd700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a84069e20 0x7f0a8406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T08:53:59.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a837fe700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a8406b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0a740078b0 tx=0x7f0a74004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:59.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a6bfff700 1 -- 172.21.15.93:0/3232751761 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a7400e030 con 0x7f0a840f6510 2021-11-20T08:53:59.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8406abb0 con 0x7f0a840f6510 2021-11-20T08:53:59.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.120+0000 7f0a6bfff700 1 -- 172.21.15.93:0/3232751761 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a74007e00 con 0x7f0a840f6510 2021-11-20T08:53:59.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.121+0000 7f0a6bfff700 1 -- 172.21.15.93:0/3232751761 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a74018070 con 0x7f0a840f6510 2021-11-20T08:53:59.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.121+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8410ecb0 con 0x7f0a840f6510 2021-11-20T08:53:59.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.122+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a84045ad0 con 0x7f0a840f6510 2021-11-20T08:53:59.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.122+0000 7f0a6bfff700 1 -- 172.21.15.93:0/3232751761 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0a740137e0 con 0x7f0a840f6510 2021-11-20T08:53:59.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.122+0000 7f0a6bfff700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0a6c05c3b0 0x7f0a6c05e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:53:59.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.122+0000 7f0a6bfff700 1 -- 172.21.15.93:0/3232751761 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a74076cd0 con 0x7f0a840f6510 2021-11-20T08:53:59.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.122+0000 7f0a82ffd700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0a6c05c3b0 0x7f0a6c05e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:53:59.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.122+0000 7f0a82ffd700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0a6c05c3b0 0x7f0a6c05e870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0a7800a820 tx=0x7f0a78008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:53:59.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.126+0000 7f0a6bfff700 1 -- 172.21.15.93:0/3232751761 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a7404a9a0 con 0x7f0a840f6510 2021-11-20T08:53:59.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.275+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0a840fbf60 con 0x7f0a840f6510 2021-11-20T08:53:59.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.276+0000 7f0a6bfff700 1 -- 172.21.15.93:0/3232751761 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+225 (secure 0 0 0) 0x7f0a7404dfc0 con 0x7f0a840f6510 2021-11-20T08:53:59.277 INFO:teuthology.orchestra.run.smithi093.stdout:# minimal ceph.conf for 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:53:59.277 INFO:teuthology.orchestra.run.smithi093.stdout:[global] 2021-11-20T08:53:59.277 INFO:teuthology.orchestra.run.smithi093.stdout: fsid = 11458b34-49df-11ec-8c2c-001a4aab830c 2021-11-20T08:53:59.277 INFO:teuthology.orchestra.run.smithi093.stdout: mon_host = [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] 2021-11-20T08:53:59.278 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0a6c05c3b0 msgr2=0x7f0a6c05e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:59.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0a6c05c3b0 0x7f0a6c05e870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0a7800a820 tx=0x7f0a78008040).stop 2021-11-20T08:53:59.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 msgr2=0x7f0a8406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:53:59.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a8406b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0a740078b0 tx=0x7f0a74004d10).stop 2021-11-20T08:53:59.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 shutdown_connections 2021-11-20T08:53:59.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0a6c05c3b0 0x7f0a6c05e870 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:59.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a840f6510 0x7f0a8406b9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:59.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 --2- 172.21.15.93:0/3232751761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a84069e20 0x7f0a8406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:53:59.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 >> 172.21.15.93:0/3232751761 conn(0x7f0a840f1c70 msgr2=0x7f0a840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:53:59.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 shutdown_connections 2021-11-20T08:53:59.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:53:59.277+0000 7f0a89a94700 1 -- 172.21.15.93:0/3232751761 wait complete. 2021-11-20T08:53:59.388 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:53:59 smithi093 conmon[33522]: audit 2021-11-20T08:53:59.276862+0000 mon.smithi093 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.93:0/3232751761' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:53:59.580 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-11-20T08:53:59.581 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:53:59.581 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T08:53:59.583 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:53:59 smithi102 conmon[36472]: audit 2021-11-20T08:53:59.276862+0000 mon.smithi093 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.93:0/3232751761' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:53:59.611 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:53:59.612 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T08:53:59.677 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:53:59.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T08:53:59.708 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:53:59.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T08:53:59.775 INFO:tasks.cephadm:Deploying OSDs... 2021-11-20T08:53:59.776 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:53:59.776 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T08:53:59.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T08:53:59.795 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T08:53:59.852 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T08:53:59.852 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:53:59.852 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 6h/6d Inode: 113636 Links: 1 2021-11-20T08:53:59.853 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:53:59.853 INFO:teuthology.orchestra.run.smithi093.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:53:59.853 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2021-11-20 08:53:29.727634536 +0000 2021-11-20T08:53:59.853 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2021-11-20 08:47:35.219703131 +0000 2021-11-20T08:53:59.854 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2021-11-20 08:47:35.219703131 +0000 2021-11-20T08:53:59.854 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2021-11-20T08:53:59.862 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2021-11-20T08:53:59.862 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2021-11-20T08:53:59.862 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000235311 s, 2.2 MB/s 2021-11-20T08:53:59.865 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T08:53:59.921 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T08:53:59.921 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:53:59.921 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 6h/6d Inode: 112599 Links: 1 2021-11-20T08:53:59.922 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:53:59.922 INFO:teuthology.orchestra.run.smithi093.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:53:59.922 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2021-11-20 08:53:29.727634536 +0000 2021-11-20T08:53:59.922 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2021-11-20 08:47:35.559695786 +0000 2021-11-20T08:53:59.923 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2021-11-20 08:47:35.559695786 +0000 2021-11-20T08:53:59.923 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2021-11-20T08:53:59.930 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2021-11-20T08:53:59.932 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2021-11-20T08:53:59.933 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000219616 s, 2.3 MB/s 2021-11-20T08:53:59.935 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T08:53:59.995 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T08:53:59.995 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:53:59.996 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 6h/6d Inode: 118123 Links: 1 2021-11-20T08:53:59.996 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:53:59.996 INFO:teuthology.orchestra.run.smithi093.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:53:59.997 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2021-11-20 08:53:29.727634536 +0000 2021-11-20T08:53:59.997 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2021-11-20 08:47:35.902688377 +0000 2021-11-20T08:53:59.997 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2021-11-20 08:47:35.902688377 +0000 2021-11-20T08:53:59.997 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2021-11-20T08:54:00.004 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2021-11-20T08:54:00.004 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2021-11-20T08:54:00.004 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000225911 s, 2.3 MB/s 2021-11-20T08:54:00.007 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T08:54:00.072 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T08:54:00.073 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:54:00.073 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 6h/6d Inode: 119983 Links: 1 2021-11-20T08:54:00.074 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:54:00.074 INFO:teuthology.orchestra.run.smithi093.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:54:00.075 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2021-11-20 08:53:29.727634536 +0000 2021-11-20T08:54:00.075 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2021-11-20 08:47:36.244680989 +0000 2021-11-20T08:54:00.076 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2021-11-20 08:47:36.244680989 +0000 2021-11-20T08:54:00.076 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2021-11-20T08:54:00.082 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2021-11-20T08:54:00.083 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2021-11-20T08:54:00.083 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.00018822 s, 2.7 MB/s 2021-11-20T08:54:00.085 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:54:00.085 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T08:54:00.106 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T08:54:00.107 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T08:54:00.163 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T08:54:00.163 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:54:00.164 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 117855 Links: 1 2021-11-20T08:54:00.164 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:54:00.164 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:54:00.164 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-11-20 08:53:57.254880254 +0000 2021-11-20T08:54:00.165 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-11-20 08:47:35.233455847 +0000 2021-11-20T08:54:00.165 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-11-20 08:47:35.233455847 +0000 2021-11-20T08:54:00.165 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-11-20T08:54:00.173 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-11-20T08:54:00.173 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-11-20T08:54:00.174 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000212265 s, 2.4 MB/s 2021-11-20T08:54:00.176 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T08:54:00.232 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T08:54:00.232 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:54:00.232 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 120841 Links: 1 2021-11-20T08:54:00.232 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:54:00.233 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:54:00.233 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-11-20 08:53:57.254880254 +0000 2021-11-20T08:54:00.233 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-11-20 08:47:35.590448641 +0000 2021-11-20T08:54:00.233 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-11-20 08:47:35.590448641 +0000 2021-11-20T08:54:00.234 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-11-20T08:54:00.242 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-11-20T08:54:00.242 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-11-20T08:54:00.243 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.00020263 s, 2.5 MB/s 2021-11-20T08:54:00.245 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T08:54:00.301 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T08:54:00.302 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:54:00.302 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 118997 Links: 1 2021-11-20T08:54:00.302 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:54:00.302 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:54:00.303 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-11-20 08:53:57.254880254 +0000 2021-11-20T08:54:00.303 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-11-20 08:47:35.953441313 +0000 2021-11-20T08:54:00.303 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-11-20 08:47:35.953441313 +0000 2021-11-20T08:54:00.303 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-11-20T08:54:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-11-20T08:54:00.311 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-11-20T08:54:00.312 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000209399 s, 2.4 MB/s 2021-11-20T08:54:00.314 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T08:54:00.372 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T08:54:00.372 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T08:54:00.373 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 117099 Links: 1 2021-11-20T08:54:00.373 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T08:54:00.373 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T08:54:00.373 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2021-11-20 08:53:57.254880254 +0000 2021-11-20T08:54:00.374 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2021-11-20 08:47:36.300434308 +0000 2021-11-20T08:54:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2021-11-20 08:47:36.300434308 +0000 2021-11-20T08:54:00.375 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2021-11-20T08:54:00.382 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2021-11-20T08:54:00.383 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2021-11-20T08:54:00.383 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.00016743 s, 3.1 MB/s 2021-11-20T08:54:00.386 INFO:tasks.cephadm:Deploying osd.0 on smithi093 with /dev/vg_nvme/lv_4... 2021-11-20T08:54:00.387 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T08:54:00.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:00 smithi102 conmon[36472]: cluster 2021-11-20T08:53:59.493492+0000 mgr.smithi093.qdikkm (mgr.14162) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:00.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:00 smithi102 conmon[36472]: audit 2021-11-20T08:53:59.666900+0000 mon.smithi093 (mon.0) 185 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:00.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:00 smithi102 conmon[36472]: audit 2021-11-20T08:53:59.667777+0000 mon.smithi093 (mon.0) 186 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:00.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:00 smithi093 conmon[33522]: cluster 2021-11-20T08:53:59.493492+0000 mgr.smithi093.qdikkm (mgr.14162) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:00.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:00 smithi093 conmon[33522]: audit 2021-11-20T08:53:59.666900+0000 mon.smithi093 (mon.0) 185 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:00.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:00 smithi093 conmon[33522]: audit 2021-11-20T08:53:59.667777+0000 mon.smithi093 (mon.0) 186 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:02.106 INFO:teuthology.orchestra.run.smithi093.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T08:54:02.106 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T08:54:02.107 INFO:teuthology.orchestra.run.smithi093.stderr: stderr: 10+0 records in 2021-11-20T08:54:02.107 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records out 2021-11-20T08:54:02.107 INFO:teuthology.orchestra.run.smithi093.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109158 s, 961 MB/s 2021-11-20T08:54:02.107 INFO:teuthology.orchestra.run.smithi093.stderr: stderr: 2021-11-20T08:54:02.108 INFO:teuthology.orchestra.run.smithi093.stderr:--> Zapping successful for: 2021-11-20T08:54:02.499 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi093:vg_nvme/lv_4 2021-11-20T08:54:02.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:02 smithi102 conmon[36472]: cluster 2021-11-20T08:54:01.493725+0000 mgr.smithi093.qdikkm (mgr.14162) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:02.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:02 smithi102 conmon[36472]: audit 2021-11-20T08:54:01.816967+0000 mon.smithi093 (mon.0) 187 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:02.951 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:02 smithi093 conmon[33522]: cluster 2021-11-20T08:54:01.493725+0000 mgr.smithi093.qdikkm (mgr.14162) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:02.951 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:02 smithi093 conmon[33522]: audit 2021-11-20T08:54:01.816967+0000 mon.smithi093 (mon.0) 187 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:03.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:03 smithi093 conmon[33522]: cluster 2021-11-20T08:54:03.494003+0000 mgr.smithi093.qdikkm (mgr.14162) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:03.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:03 smithi102 conmon[36472]: cluster 2021-11-20T08:54:03.494003+0000 2021-11-20T08:54:03.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:03 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:04.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 -- 172.21.15.93:0/1397313385 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f7d40 msgr2=0x7f61300f8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:04.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 --2- 172.21.15.93:0/1397313385 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f7d40 0x7f61300f8160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f6120004660 tx=0x7f6120009b30).stop 2021-11-20T08:54:04.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 -- 172.21.15.93:0/1397313385 shutdown_connections 2021-11-20T08:54:04.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 --2- 172.21.15.93:0/1397313385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61300f86e0 0x7f61300f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:54:04.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 --2- 172.21.15.93:0/1397313385 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f7d40 0x7f61300f8160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:04.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 -- 172.21.15.93:0/1397313385 >> 172.21.15.93:0/1397313385 conn(0x7f61300f04c0 msgr2=0x7f61300f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:54:04.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 -- 172.21.15.93:0/1397313385 shutdown_connections 2021-11-20T08:54:04.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 -- 172.21.15.93:0/1397313385 wait complete. 2021-11-20T08:54:04.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 Processor -- start 2021-11-20T08:54:04.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.013+0000 7f6135df5700 1 -- start start 2021-11-20T08:54:04.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6135df5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61300f7d40 0x7f6130107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:04.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6135df5700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f86e0 0x7f6130107500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:04.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6135df5700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6130107b40 con 0x7f61300f86e0 2021-11-20T08:54:04.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6135df5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61301083f0 con 0x7f61300f7d40 2021-11-20T08:54:04.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f86e0 0x7f6130107500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:04.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f86e0 0x7f6130107500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42526/0 (socket says 172.21.15.93:42526) 2021-11-20T08:54:04.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612effd700 1 -- 172.21.15.93:0/2509761930 learned_addr learned my addr 172.21.15.93:0/2509761930 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:54:04.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612f7fe700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61300f7d40 0x7f6130107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:04.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612effd700 1 -- 172.21.15.93:0/2509761930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61300f7d40 msgr2=0x7f6130107010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:04.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612effd700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61300f7d40 0x7f6130107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:04.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612effd700 1 -- 172.21.15.93:0/2509761930 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6120005040 con 0x7f61300f86e0 2021-11-20T08:54:04.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f612effd700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f86e0 0x7f6130107500 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f612400b410 tx=0x7f6124006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:04.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6117fff700 1 -- 172.21.15.93:0/2509761930 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61240125f0 con 0x7f61300f86e0 2021-11-20T08:54:04.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6117fff700 1 -- 172.21.15.93:0/2509761930 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6124012c00 con 0x7f61300f86e0 2021-11-20T08:54:04.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6117fff700 1 -- 172.21.15.93:0/2509761930 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6124002aa0 con 0x7f61300f86e0 2021-11-20T08:54:04.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61301086d0 con 0x7f61300f86e0 2021-11-20T08:54:04.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.014+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6130108c90 con 0x7f61300f86e0 2021-11-20T08:54:04.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.015+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61300f4150 con 0x7f61300f86e0 2021-11-20T08:54:04.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.015+0000 7f6117fff700 1 -- 172.21.15.93:0/2509761930 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f612400f030 con 0x7f61300f86e0 2021-11-20T08:54:04.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.016+0000 7f6117fff700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f611805c2b0 0x7f611805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:04.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.016+0000 7f6117fff700 1 -- 172.21.15.93:0/2509761930 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6124075370 con 0x7f61300f86e0 2021-11-20T08:54:04.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.016+0000 7f612f7fe700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f611805c2b0 0x7f611805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:04.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.016+0000 7f612f7fe700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f611805c2b0 0x7f611805e770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6120009d60 tx=0x7f6120009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:04.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.020+0000 7f6117fff700 1 -- 172.21.15.93:0/2509761930 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6124049040 con 0x7f61300f86e0 2021-11-20T08:54:04.206 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:04.206+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f61300fef70 con 0x7f611805c2b0 2021-11-20T08:54:04.850 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:04 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.209326 2021-11-20T08:54:04.851 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:04 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 47 : audit [DBG] from='client.14212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:04.852 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:04 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.210304+0000 mon.smithi093 (mon.0) 188 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:04.852 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:04 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:04.211562+0000 mgr.smithi093.qdikkm (mgr.14162) 48 : cephadm [INF] Applying service osd.None on host smithi093... 2021-11-20T08:54:04.852 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:04 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.211761+0000 mon.smithi093 (mon.0) 189 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:04.853 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:04 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.212282+0000 mon.smithi093 (mon.0) 190 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:04.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: audit 2021-11-20T08:54:04.209326+0000 2021-11-20T08:54:04.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 47 : audit [DBG] from='client.14212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: audit 2021-11-20T08:54:04.210304 2021-11-20T08:54:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 188 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:04 2021-11-20T08:54:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: .211562+0000 mgr.smithi093.qdikkm (mgr.14162) 48 : cephadm [INF] 2021-11-20T08:54:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: Applying service osd.None on host smithi093... 2021-11-20T08:54:04.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: audit 2021-11-20T08:54: 2021-11-20T08:54:04.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: 04.211761+0000 mon.smithi093 (mon.0) 189 : audit 2021-11-20T08:54:04.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:04.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: audit 2021-11-20T08:54: 2021-11-20T08:54:04.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: 04.212282+0000 mon.smithi093 (mon.0) 190 : audit 2021-11-20T08:54:04.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:04 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:05.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: audit 2021-11-20T08:54:04.574040 2021-11-20T08:54:05.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 191 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:05.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T08:54:05.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: 08:54:04.957378+0000 mon.smithi093 (mon.0) 192 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:05.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: cephadm 2021 2021-11-20T08:54:05.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: -11-20T08:54:04.959310+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:54:05.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: .14162) 49 : cephadm [INF] Reconfiguring mon.smithi093 (unknown last config time)... 2021-11-20T08:54:05.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:05.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: -11-20T08:54:04.959548+0000 mon.smithi093 (mon.0 2021-11-20T08:54:05.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: ) 193 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T08:54:05.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:05.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: -11-20T08:54:04.960110+0000 mon.smithi093 ( 2021-11-20T08:54:05.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: mon.0) 194 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T08:54:05.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: audit 2021-11-20T08: 2021-11-20T08:54:05.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: 54:04.960608+0000 mon.smithi093 (mon.0) 195 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:05.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: 2021-11-20T08:54:05.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: cephadm 2021-11-20T08: 2021-11-20T08:54:05.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: 54:04.961056+0000 mgr.smithi093.qdikkm (mgr.14162) 50 : 2021-11-20T08:54:05.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: cephadm [INF] Reconfiguring daemon mon.smithi093 on smithi093 2021-11-20T08:54:05.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:54:05.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: -11-20T08:54:05.494378+0000 mgr.smithi093.qdikkm 2021-11-20T08:54:05.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:05 smithi102 conmon[36472]: (mgr.14162) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.574040+0000 mon.smithi093 (mon.0) 191 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.957378+0000 mon.smithi093 (mon.0) 192 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:04.959310+0000 mgr.smithi093.qdikkm (mgr.14162) 49 : cephadm [INF] Reconfiguring mon.smithi093 (unknown last config time)... 2021-11-20T08:54:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.959548+0000 mon.smithi093 (mon.0) 193 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T08:54:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.960110+0000 mon.smithi093 (mon.0) 194 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T08:54:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: audit 2021-11-20T08:54:04.960608+0000 mon.smithi093 (mon.0) 195 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:04.961056+0000 mgr.smithi093.qdikkm (mgr.14162) 50 : cephadm [INF] Reconfiguring daemon mon.smithi093 on smithi093 2021-11-20T08:54:05.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:05 smithi093 conmon[33522]: cluster 2021-11-20T08:54:05.494378+0000 mgr.smithi093.qdikkm (mgr.14162) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:07.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: audit 2021-11-20T08:54:06.333416+0000 mon.smithi093 (mon.0) 196 2021-11-20T08:54:07.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: : audit [INF] from='client.? 172.21.15.93:0/136570091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b62eaab5-818e-4739-98c3-d386425d4837"}]: dispatch 2021-11-20T08:54:07.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: audit 2021-11-20T08:54:06.337145+0000 mon.smithi093 (mon.0) 197 : audit [INF] from='client.? 172.21.15.93:0/136570091' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b62eaab5-818e-4739-98c3-d386425d4837"}]': finished 2021-11-20T08:54:07.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: cluster 2021-11-20T08:54:06.337186+0000 2021-11-20T08:54:07.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: mon.smithi093 (mon.0) 198 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T08:54:07.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:54:07.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: 11-20T08:54:06.337289+0000 mon.smithi093 (mon.0 2021-11-20T08:54:07.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: ) 199 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:07.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: audit 2021-11-20T08:54: 2021-11-20T08:54:07.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: 06.742090+0000 mon.smithi093 (mon.0) 200 : audit [DBG] 2021-11-20T08:54:07.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:07 smithi102 conmon[36472]: from='client.? 172.21.15.93:0/2249739381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:54:07.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:07 smithi093 conmon[33522]: audit 2021-11-20T08:54:06.333416+0000 mon.smithi093 ( 2021-11-20T08:54:07.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:07 smithi093 conmon[33522]: mon.0) 196 : audit [INF] from='client.? 172.21.15.93:0/136570091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b62eaab5-818e-4739-98c3-d386425d4837"}]: dispatch 2021-11-20T08:54:07.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:07 smithi093 conmon[33522]: audit 2021-11-20T08:54:06.337145+0000 mon.smithi093 (mon.0) 197 : audit [INF] from='client.? 172.21.15.93:0/136570091' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b62eaab5-818e-4739-98c3-d386425d4837"}]': finished 2021-11-20T08:54:07.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:07 smithi093 conmon[33522]: cluster 2021-11-20T08:54:06.337186+0000 mon.smithi093 (mon.0) 198 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T08:54:07.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:07 smithi093 conmon[33522]: audit 2021-11-20T08:54:06.337289+0000 mon.smithi093 (mon.0) 199 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:07.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:07 smithi093 conmon[33522]: audit 2021-11-20T08:54:06.742090+0000 mon.smithi093 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.93:0/2249739381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:54:08.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:08 smithi102 conmon[36472]: cluster 2021-11-20T08:54:07.494876+0000 2021-11-20T08:54:08.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:08 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 52 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:08.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:08 smithi093 conmon[33522]: cluster 2021-11-20T08:54:07.494876 2021-11-20T08:54:08.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:08 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 52 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:09.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:09 smithi102 conmon[36472]: cluster 2021-11-20T08:54:09.495196+0000 mgr.smithi093.qdikkm (mgr.14162) 53 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:09.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:09 smithi093 conmon[33522]: cluster 2021-11-20T08:54:09.495196+0000 mgr.smithi093.qdikkm (mgr.14162) 53 2021-11-20T08:54:09.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:09 smithi093 conmon[33522]: : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:12.640 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: audit 2021-11-20T08:54:11.297185+0000 mon.smithi093 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:12.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:11.297636+0000 mgr.smithi093.qdikkm (mgr.14162) 54 : cephadm [INF] Reconfiguring mgr.smithi093.qdikkm (unknown last config time)... 2021-11-20T08:54:12.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: audit 2021-11-20T08:54:11.297866+0000 mon.smithi093 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi093.qdikkm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T08:54:12.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: audit 2021-11-20T08:54:11.298497+0000 mon.smithi093 (mon.0) 203 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T08:54:12.641 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: audit 2021-11-20T08:54:11.298999+0000 mon.smithi093 (mon.0) 204 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:12.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:11.299459+0000 mgr.smithi093.qdikkm (mgr.14162) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi093.qdikkm on smithi093 2021-11-20T08:54:12.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: cluster 2021-11-20T08:54:12.642 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:12 smithi093 conmon[33522]: 2021-11-20T08:54:11.495574+0000 mgr.smithi093.qdikkm (mgr.14162) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:12.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: audit 2021-11-20T08:54:11.297185+0000 mon.smithi093 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:12.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: cephadm 2021-11-20T08:54: 2021-11-20T08:54:12.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: 11.297636+0000 mgr.smithi093.qdikkm (mgr.14162) 54 : cephadm [INF] Reconfiguring mgr.smithi093.qdikkm (unknown last config time)... 2021-11-20T08:54:12.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: audit 2021-11-20T08:54:11.297866+0000 mon.smithi093 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi093.qdikkm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T08:54:12.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: audit 2021-11-20T08:54:11.298497+0000 mon.smithi093 2021-11-20T08:54:12.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: (mon.0) 203 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T08:54:12.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: 2021-11-20T08:54:12.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: audit 2021-11-20T08:54:12.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: 2021-11-20T08:54: 2021-11-20T08:54:12.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: 11.298999+0000 mon.smithi093 (mon.0 2021-11-20T08:54:12.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: ) 204 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:12.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:11.299459+0000 mgr.smithi093.qdikkm (mgr.14162) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi093.qdikkm on smithi093 2021-11-20T08:54:12.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: cluster 2021-11-20T08:54:11.495574+0000 mgr.smithi093.qdikkm (mgr.14162) 56 : cluster [DBG] 2021-11-20T08:54:12.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:12 smithi102 conmon[36472]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:13.750 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:13 smithi093 conmon[33522]: audit 2021-11-20T08:54:12.644295+0000 mon.smithi093 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T08:54:13.750 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:13 smithi093 conmon[33522]: audit 2021-11-20T08:54:12.644946+0000 mon.smithi093 (mon.0) 206 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:13.750 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:13 smithi093 conmon[33522]: audit 2021-11-20T08:54:13.441500+0000 mon.smithi093 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:13.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: audit 2021-11-20T08:54:12.644295+0000 mon.smithi093 ( 2021-11-20T08:54:13.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T08:54:13.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: audit 2021-11-20T08:54:12.644946+0000 mon.smithi093 (mon.0) 206 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:13.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: 2021-11-20T08:54:13.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: audit 2021-11-20T08:54:13.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: 2021-11-20T08:54:13. 2021-11-20T08:54:13.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: 441500+0000 mon.smithi093 (mon.0) 207 2021-11-20T08:54:13.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:13 smithi102 conmon[36472]: : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:14.807 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:14 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:12.645390+0000 mgr.smithi093.qdikkm (mgr.14162) 57 : cephadm [INF] Deploying daemon osd.0 on smithi093 2021-11-20T08:54:14.807 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:14 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:13.442102+0000 mgr.smithi093.qdikkm (mgr.14162) 58 : cephadm [INF] Reconfiguring alertmanager.smithi093 (dependencies changed)... 2021-11-20T08:54:14.807 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:14 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:13.444299+0000 mgr.smithi093.qdikkm (mgr.14162) 59 : cephadm [INF] Reconfiguring daemon alertmanager.smithi093 on smithi093 2021-11-20T08:54:14.808 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:14 smithi093 conmon[33522]: cluster 2021-11-20T08:54:13.495862+0000 mgr.smithi093.qdikkm (mgr.14162) 60 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:12.645390+0000 2021-11-20T08:54:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 57 : cephadm [INF] Deploying daemon osd.0 on smithi093 2021-11-20T08:54:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:13.442102+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:54:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: .14162) 58 : cephadm [INF] Reconfiguring alertmanager.smithi093 (dependencies changed)... 2021-11-20T08:54:14.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:14.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: 2021-11-20T08:54: 2021-11-20T08:54:14.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: 13.444299+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:14.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: 14162) 59 : cephadm [INF] Reconfiguring daemon alertmanager.smithi093 on smithi093 2021-11-20T08:54:14.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: cluster 2021- 2021-11-20T08:54:14.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: 11-20T08:54:13.495862+0000 2021-11-20T08:54:14.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 60 : cluster 2021-11-20T08:54:14.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:14 smithi102 conmon[36472]: [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:15.890 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:15 smithi093 conmon[33522]: cluster 2021-11-20T08:54:15.496040+0000 mgr.smithi093.qdikkm (mgr.14162) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:15.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:15 smithi102 conmon[36472]: cluster 2021-11-20T08:54:15.496040+0000 mgr.smithi093.qdikkm (mgr.14162) 61 : cluster 2021-11-20T08:54:15.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:15 smithi102 conmon[36472]: [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:16.053 INFO:teuthology.orchestra.run.smithi093.stdout:Created osd(s) 0 on host 'smithi093' 2021-11-20T08:54:16.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.052+0000 7f6117fff700 1 -- 172.21.15.93:0/2509761930 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f61300fef70 con 0x7f611805c2b0 2021-11-20T08:54:16.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.055+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f611805c2b0 msgr2=0x7f611805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:16.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.055+0000 7f6135df5700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f611805c2b0 0x7f611805e770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6120009d60 tx=0x7f6120009b30).stop 2021-11-20T08:54:16.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.055+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f86e0 msgr2=0x7f6130107500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:16.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.055+0000 7f6135df5700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f86e0 0x7f6130107500 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f612400b410 tx=0x7f6124006ef0).stop 2021-11-20T08:54:16.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.056+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 shutdown_connections 2021-11-20T08:54:16.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.056+0000 7f6135df5700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f611805c2b0 0x7f611805e770 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:16.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.056+0000 7f6135df5700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61300f7d40 0x7f6130107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:16.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.056+0000 7f6135df5700 1 --2- 172.21.15.93:0/2509761930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61300f86e0 0x7f6130107500 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:16.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.056+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 >> 172.21.15.93:0/2509761930 conn(0x7f61300f04c0 msgr2=0x7f61300fb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:54:16.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.057+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 shutdown_connections 2021-11-20T08:54:16.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:16.057+0000 7f6135df5700 1 -- 172.21.15.93:0/2509761930 wait complete. 2021-11-20T08:54:16.421 DEBUG:teuthology.orchestra.run.smithi093:osd.0> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.0.service 2021-11-20T08:54:16.424 INFO:tasks.cephadm:Deploying osd.1 on smithi093 with /dev/vg_nvme/lv_3... 2021-11-20T08:54:16.424 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T08:54:16.453 INFO:journalctl@ceph.osd.0.smithi093.stdout:-- Logs begin at Sat 2021-11-20 08:41:08 UTC. -- 2021-11-20T08:54:16.730 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 08:54:16 smithi093 conmon[49856]: debug 2021-11-20T08:54:16.516+0000 7fd9313ee080 -1 Falling back to public interface 2021-11-20T08:54:17.274 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:17 smithi093 conmon[33522]: audit 2021-11-20T08:54:16.048445+0000 mon.smithi093 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:17.274 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:17 smithi093 conmon[33522]: audit 2021-11-20T08:54:16.052978+0000 mon.smithi093 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:17.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:17 smithi102 conmon[36472]: audit 2021-11-20T08:54:16.048445+0000 mon.smithi093 (mon.0 2021-11-20T08:54:17.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:17 smithi102 conmon[36472]: ) 208 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:17.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:17 smithi102 conmon[36472]: audit 2021-11-20T08:54:16.052978+0000 mon.smithi093 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:18.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:18 smithi093 conmon[33522]: cluster 2021-11-20T08:54:17.496290+0000 mgr.smithi093.qdikkm (mgr.14162) 62 : cluster 2021-11-20T08:54:18.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:18 smithi093 conmon[33522]: [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:18.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:18 smithi102 conmon[36472]: cluster 2021-11-20T08:54:17.496290+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:54:18.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:18 smithi102 conmon[36472]: mgr.14162) 62 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:19.176 INFO:teuthology.orchestra.run.smithi093.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T08:54:19.177 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T08:54:19.177 INFO:teuthology.orchestra.run.smithi093.stderr: stderr: 10+0 records in 2021-11-20T08:54:19.177 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records out 2021-11-20T08:54:19.177 INFO:teuthology.orchestra.run.smithi093.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138482 s, 757 MB/s 2021-11-20T08:54:19.178 INFO:teuthology.orchestra.run.smithi093.stderr:--> Zapping successful for: 2021-11-20T08:54:19.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi093:vg_nvme/lv_3 2021-11-20T08:54:19.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:19 smithi102 conmon[36472]: cluster 2021-11-20T08:54:19.496491+0000 mgr.smithi093.qdikkm 2021-11-20T08:54:19.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:19 smithi102 conmon[36472]: (mgr.14162) 63 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:19.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:19 smithi093 conmon[33522]: cluster 2021-11-20T08:54:19.496491+0000 2021-11-20T08:54:19.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:19 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 63 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:20.290 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 08:54:20 smithi093 conmon[49856]: debug 2021-11-20T08:54:20.040+0000 7fd9313ee080 -1 osd.0 0 log_to_monitors {default=true} 2021-11-20T08:54:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:20 smithi093 conmon[33522]: audit 2021-11-20T08:54:19.672226+0000 mon.smithi093 (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:20 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:19.672823+0000 mgr.smithi093.qdikkm (mgr.14162) 64 : cephadm [INF] Reconfiguring crash.smithi093 (monmap changed)... 2021-11-20T08:54:20.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:20 smithi093 conmon[33522]: audit 2021-11-20T08:54:19.673069+0000 mon.smithi093 (mon.0) 211 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi093", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T08:54:20.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:20 smithi093 conmon[33522]: audit 2021-11-20T08:54:19.673777+0000 mon.smithi093 (mon.0) 212 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:20.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:20 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:19.674403+0000 mgr.smithi093.qdikkm (mgr.14162) 65 : cephadm [INF] Reconfiguring daemon crash.smithi093 on smithi093 2021-11-20T08:54:20.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:20 smithi093 conmon[33522]: audit 2021-11-20T08:54:20.044912+0000 mon.smithi093 (mon.0) 213 : audit [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T08:54:21.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: audit 2021-11-20T08:54:19. 2021-11-20T08:54:21.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: 672226+0000 mon.smithi093 (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:21.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:19.672823+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:54:21.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: ) 64 : cephadm [INF] Reconfiguring crash.smithi093 (monmap changed)... 2021-11-20T08:54:21.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: audit 2021-11-20T08 2021-11-20T08:54:21.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: :54:19.673069+0000 mon.smithi093 (mon.0) 211 : audit [INF] 2021-11-20T08:54:21.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi093", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T08:54:21.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: audit 2021-11-20T08:54:21.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: 2021-11-20T08:54:19.673777+0000 mon.smithi093 (mon.0 2021-11-20T08:54:21.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: ) 212 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:21.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: cephadm 2021-11-20 2021-11-20T08:54:21.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: T08:54:19.674403+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:54:21.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: .14162) 65 : cephadm [INF] Reconfiguring daemon crash.smithi093 on smithi093 2021-11-20T08:54:21.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:54:21.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: 11-20T08:54:20.044912+0000 mon.smithi093 ( 2021-11-20T08:54:21.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:20 smithi102 conmon[36472]: mon.0) 213 : audit [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T08:54:21.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 -- 172.21.15.93:0/1021426842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 msgr2=0x7f7f1c0f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:21.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 --2- 172.21.15.93:0/1021426842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c0f7150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f7f10004660 tx=0x7f7f10009b30).stop 2021-11-20T08:54:21.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 -- 172.21.15.93:0/1021426842 shutdown_connections 2021-11-20T08:54:21.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 --2- 172.21.15.93:0/1021426842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c0f7640 0x7f7f1c0f9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:54:21.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 --2- 172.21.15.93:0/1021426842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c0f7150 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:21.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 -- 172.21.15.93:0/1021426842 >> 172.21.15.93:0/1021426842 conn(0x7f7f1c0f0de0 msgr2=0x7f7f1c0f3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:54:21.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 -- 172.21.15.93:0/1021426842 shutdown_connections 2021-11-20T08:54:21.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 -- 172.21.15.93:0/1021426842 wait complete. 2021-11-20T08:54:21.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.441+0000 7f7f23406700 1 Processor -- start 2021-11-20T08:54:21.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f23406700 1 -- start start 2021-11-20T08:54:21.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f23406700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c10d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:21.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f23406700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c0f7640 0x7f7f1c10dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:21.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f23406700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c10e110 con 0x7f7f1c060390 2021-11-20T08:54:21.455 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f23406700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c10e9c0 con 0x7f7f1c0f7640 2021-11-20T08:54:21.455 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f211a2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c10d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:21.456 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f211a2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c10d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42564/0 (socket says 172.21.15.93:42564) 2021-11-20T08:54:21.458 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f211a2700 1 -- 172.21.15.93:0/1152886602 learned_addr learned my addr 172.21.15.93:0/1152886602 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:54:21.458 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f209a1700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c0f7640 0x7f7f1c10dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:21.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f211a2700 1 -- 172.21.15.93:0/1152886602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c0f7640 msgr2=0x7f7f1c10dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:21.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f211a2700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c0f7640 0x7f7f1c10dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:21.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f211a2700 1 -- 172.21.15.93:0/1152886602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f10005040 con 0x7f7f1c060390 2021-11-20T08:54:21.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f211a2700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c10d5e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f7f10009f50 tx=0x7f7f10009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:21.461 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f0dffb700 1 -- 172.21.15.93:0/1152886602 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f10018070 con 0x7f7f1c060390 2021-11-20T08:54:21.462 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c10ec40 con 0x7f7f1c060390 2021-11-20T08:54:21.462 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f0dffb700 1 -- 172.21.15.93:0/1152886602 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f10013410 con 0x7f7f1c060390 2021-11-20T08:54:21.462 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.442+0000 7f7f0dffb700 1 -- 172.21.15.93:0/1152886602 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f1001c620 con 0x7f7f1c060390 2021-11-20T08:54:21.463 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.443+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c10f1a0 con 0x7f7f1c060390 2021-11-20T08:54:21.463 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.443+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c0f4a70 con 0x7f7f1c060390 2021-11-20T08:54:21.463 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.444+0000 7f7f0dffb700 1 -- 172.21.15.93:0/1152886602 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7f1001d420 con 0x7f7f1c060390 2021-11-20T08:54:21.463 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.444+0000 7f7f0dffb700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7f0805c2b0 0x7f7f0805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:21.464 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.444+0000 7f7f0dffb700 1 -- 172.21.15.93:0/1152886602 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1476+0+0 (secure 0 0 0) 0x7f7f10076d70 con 0x7f7f1c060390 2021-11-20T08:54:21.464 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.444+0000 7f7f209a1700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7f0805c2b0 0x7f7f0805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:21.464 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.445+0000 7f7f209a1700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7f0805c2b0 0x7f7f0805e770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7f1800b790 tx=0x7f7f18008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:21.464 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.448+0000 7f7f0dffb700 1 -- 172.21.15.93:0/1152886602 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f1004a9f0 con 0x7f7f1c060390 2021-11-20T08:54:21.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:21.639+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f1c105510 con 0x7f7f0805c2b0 2021-11-20T08:54:21.701 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[49856]: debug 2021-11-20T08:54:21.685+0000 7fd92795f700 -1 osd.0 0 waiting for initial osdmap 2021-11-20T08:54:21.701 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[49856]: debug 2021-11-20T08:54:21.689+0000 7fd9220d7700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T08:54:21.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: audit 2021-11-20T08:54:20.677177+0000 mon.smithi093 (mon.0) 214 : audit [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T08:54:21.702 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: cluster 2021-11-20T08:54:21.702 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 2021-11-20T08:54:20.677221+0000 mon.smithi093 (mon.0) 215 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T08:54:21.703 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: audit 2021-11-20T08:54:20.677350+0000 mon.smithi093 (mon.0) 216 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:21.703 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: audit 2021-11-20T08:54:20.677670 2021-11-20T08:54:21.704 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 217 : audit 2021-11-20T08:54:21.704 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]: dispatch 2021-11-20T08:54:21.705 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:54:21.705 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: -20T08:54:21.210048 2021-11-20T08:54:21.705 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 218 : audit [INF] 2021-11-20T08:54:21.705 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:21.706 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: cephadm 2021-11-20 2021-11-20T08:54:21.706 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: T08:54:21.210623+0000 2021-11-20T08:54:21.707 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 66 : cephadm [INF] 2021-11-20T08:54:21.707 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: Reconfiguring grafana.smithi093 (dependencies changed)... 2021-11-20T08:54:21.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: cephadm 2021-11- 2021-11-20T08:54:21.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 20T08:54:21.215615 2021-11-20T08:54:21.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 67 : 2021-11-20T08:54:21.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: cephadm [INF] Reconfiguring daemon grafana.smithi093 on smithi093 2021-11-20T08:54:21.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:54:21.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: -11-20T08:54: 2021-11-20T08:54:21.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 21.496890+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:21.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 14162) 68 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:21.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:54:21.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 20T08:54:21.641889+0000 mon.smithi093 ( 2021-11-20T08:54:21.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: mon.0) 219 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:21.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: audit 2021 2021-11-20T08:54:21.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: -11-20T08:54: 2021-11-20T08:54:21.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 21.643191+0000 mon.smithi093 (mon. 2021-11-20T08:54:21.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 0) 220 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:21.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:54:21.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: 11-20T08:54:21.643727+0000 2021-11-20T08:54:21.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: mon.smithi093 (mon.0) 221 : audit [DBG] 2021-11-20T08:54:21.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:21 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:22.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: audit 2021-11-20T08: 2021-11-20T08:54:22.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: 54:20.677177+0000 mon.smithi093 (mon.0) 214 : audit [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T08:54:22.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: cluster 2021-11-20T08:54:20.677221+0000 mon.smithi093 (mon.0) 215 2021-11-20T08:54:22.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T08:54:22.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:54:22.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: T08:54:20.677350+0000 mon.smithi093 (mon.0) 216 : audit 2021-11-20T08:54:22.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:22.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: audit 2021-11-20T08:54:22.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: 2021-11-20T08:54:20.677670+0000 mon.smithi093 (mon. 2021-11-20T08:54:22.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: 0) 217 : audit [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]: dispatch 2021-11-20T08:54:22.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:54:22.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: 20T08:54:21.210048+0000 mon.smithi093 ( 2021-11-20T08:54:22.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: mon.0) 218 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:22.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: cephadm 2021-11-20 2021-11-20T08:54:22.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: T08:54:21.210623+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:54:22.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: .14162) 66 : cephadm [INF] Reconfiguring grafana.smithi093 (dependencies changed)... 2021-11-20T08:54:22.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: cephadm 2021-11-20T08 2021-11-20T08:54:22.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: :54:21.215615+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:54:22.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: ) 67 : cephadm [INF] Reconfiguring daemon grafana.smithi093 on smithi093 2021-11-20T08:54:22.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: cluster 2021-11-20T08:54 2021-11-20T08:54:22.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: :21.496890+0000 mgr.smithi093.qdikkm (mgr.14162) 68 2021-11-20T08:54:22.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:22.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: audit 2021-11-20T08:54:21 2021-11-20T08:54:22.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: .641889+0000 mon.smithi093 (mon.0) 219 : audit 2021-11-20T08:54:22.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:22.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: audit 2021-11-20T08:54:22.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: 2021-11-20T08:54:21.643191+0000 mon.smithi093 ( 2021-11-20T08:54:22.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: mon.0) 220 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:22.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T08:54:22.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: 08:54:21.643727+0000 mon.smithi093 (mon 2021-11-20T08:54:22.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:21 smithi102 conmon[36472]: .0) 221 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:22 smithi093 conmon[33522]: audit 2021-11-20T08:54:21.641190+0000 mgr.smithi093.qdikkm (mgr.14162) 69 : audit [DBG] from='client.14230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:23.044 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:22 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:21.643014+0000 mgr.smithi093.qdikkm (mgr.14162) 70 : cephadm [INF] Applying service osd.None on host smithi093... 2021-11-20T08:54:23.044 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:22 smithi093 conmon[33522]: audit 2021-11-20T08:54:21.682994+0000 mon.smithi093 (mon.0) 222 : audit [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]': finished 2021-11-20T08:54:23.044 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:22 smithi093 conmon[33522]: cluster 2021-11-20T08:54:21.683046+0000 mon.smithi093 (mon.0) 223 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-20T08:54:23.045 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:22 smithi093 conmon[33522]: audit 2021-11-20T08:54:21.683365+0000 mon.smithi093 (mon.0) 224 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:23.045 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:22 smithi093 conmon[33522]: audit 2021-11-20T08:54:21.687177+0000 mon.smithi093 (mon.0) 225 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:23.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: audit 2021-11-20T08:54:21.641190+0000 mgr.smithi093.qdikkm (mgr.14162) 69 : audit [DBG] from='client.14230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:23.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:21.643014+0000 mgr.smithi093.qdikkm (mgr.14162) 70 : cephadm [INF] Applying service osd.None on host smithi093... 2021-11-20T08:54:23.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: audit 2021-11-20T08:54:23.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: 2021-11-20T08:54:21.682994+0000 mon.smithi093 (mon.0) 222 : audit 2021-11-20T08:54:23.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: [INF] from='osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]': finished 2021-11-20T08:54:23.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: cluster 2021-11-20T08 2021-11-20T08:54:23.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: :54:21.683046+0000 mon.smithi093 (mon.0) 2021-11-20T08:54:23.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: 223 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-20T08:54:23.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:54:23.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: 20T08:54:21.683365+0000 mon.smithi093 ( 2021-11-20T08:54:23.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: mon.0) 224 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:23.177 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:54:23.177 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: 11-20T08:54:21.687177+0000 2021-11-20T08:54:23.177 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: mon.smithi093 (mon.0) 225 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:23.177 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:22 smithi102 conmon[36472]: 2021-11-20T08:54:23.942 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: cluster 2021-11-20T08:54 2021-11-20T08:54:23.942 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: :21.010580+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:54:23.943 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: cluster 2021-11-20T08:54:21.010673+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:54:23.943 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:54:23.944 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: 54:22.688349+0000 mon.smithi093 (mon.0) 226 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:23.944 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:54:23.945 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: 08:54:22.691268+0000 mon.smithi093 (mon. 2021-11-20T08:54:23.945 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: 0) 227 : cluster [INF] osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] boot 2021-11-20T08:54:23.946 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:54:23.946 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: -11-20T08:54:22 2021-11-20T08:54:23.947 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: .691299+0000 mon.smithi093 (mon.0) 228 : cluster 2021-11-20T08:54:23.947 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-20T08:54:23.947 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: audit 2021-11-20 2021-11-20T08:54:23.948 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: T08:54:22.691501+0000 2021-11-20T08:54:23.948 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: mon.smithi093 (mon.0) 229 : audit [DBG] 2021-11-20T08:54:23.949 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:23.949 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:54:23.950 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: 20T08:54:23.497133+0000 2021-11-20T08:54:23.950 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 71 : cluster [DBG] 2021-11-20T08:54:23.950 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:23 smithi093 conmon[33522]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:24.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: cluster 2021-11-20T08:54:21.010580+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:54:24.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:54:24.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: T08:54:21.010673+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:54:24.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: audit 2021-11-20T08:54:22.688349+0000 mon.smithi093 (mon.0) 226 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:24.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: cluster 2021-11-20T08:54:22. 2021-11-20T08:54:24.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: 691268+0000 mon.smithi093 (mon.0) 227 : cluster [INF] 2021-11-20T08:54:24.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: osd.0 [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] boot 2021-11-20T08:54:24.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: cluster 2021-11-20T08:54:22 2021-11-20T08:54:24.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: .691299+0000 mon.smithi093 (mon.0) 228 : cluster 2021-11-20T08:54:24.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-20T08:54:24.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: audit 2021-11-20T08:54: 2021-11-20T08:54:24.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: 22.691501+0000 mon.smithi093 (mon.0) 229 : 2021-11-20T08:54:24.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T08:54:24.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T08:54:24.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: 08:54:23.497133+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:24.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:23 smithi102 conmon[36472]: 14162) 71 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T08:54:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:24 smithi093 conmon[33522]: cluster 2021-11-20T08:54:23.698779+0000 mon.smithi093 (mon.0) 230 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T08:54:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:24 smithi093 conmon[33522]: audit 2021-11-20T08:54:24.251747+0000 mon.smithi093 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:24 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:24.252591+0000 mgr.smithi093.qdikkm (mgr.14162) 72 : cephadm [INF] Reconfiguring prometheus.smithi093 (dependencies changed)... 2021-11-20T08:54:24.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:24 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:24.256056+0000 mgr.smithi093.qdikkm (mgr.14162) 73 : cephadm [INF] Reconfiguring daemon prometheus.smithi093 on smithi093 2021-11-20T08:54:25.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:24 smithi102 conmon[36472]: cluster 2021-11-20T08:54:23.698779+0000 mon.smithi093 (mon.0) 230 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T08:54:25.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:24 smithi102 conmon[36472]: audit 2021-11-20T08 2021-11-20T08:54:25.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:24 smithi102 conmon[36472]: :54:24.251747+0000 mon.smithi093 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:25.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:24 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:24.252591+0000 mgr.smithi093.qdikkm (mgr.14162) 72 : cephadm [INF] Reconfiguring prometheus.smithi093 (dependencies changed)... 2021-11-20T08:54:25.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:24 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:24.256056+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:54:25.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:24 smithi102 conmon[36472]: ) 73 : cephadm [INF] Reconfiguring daemon prometheus.smithi093 on smithi093 2021-11-20T08:54:25.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: audit 2021-11-20T08:54:25.188516+0000 mon.smithi093 (mon 2021-11-20T08:54:25.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: .0) 232 : audit [INF] from='client.? 172.21.15.93:0/2646113948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a46f3336-3bc4-4736-9095-c4d72e662323"}]: dispatch 2021-11-20T08:54:25.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: audit 2021-11-20T08:54:25.192078+0000 mon.smithi093 (mon.0) 233 : audit [INF] from='client.? 172.21.15.93:0/2646113948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a46f3336-3bc4-4736-9095-c4d72e662323"}]': finished 2021-11-20T08:54:25.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: cluster 2021-11-20T08:54:25.192121+0000 mon.smithi093 (mon.0) 234 : cluster [DBG] 2021-11-20T08:54:25.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: osdmap e11: 2 total, 1 up, 2 in 2021-11-20T08:54:25.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: audit 2021-11-20T08:54:25. 2021-11-20T08:54:25.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: 192249+0000 mon.smithi093 (mon.0) 235 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: cluster 2021-11-20T08:54:25.497416+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:54:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: mgr.14162) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:54:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: -20T08:54:25.597364 2021-11-20T08:54:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:25 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.93:0/664788105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:54:26.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: audit 2021-11-20T08:54:25.188516+0000 mon.smithi093 (mon.0) 232 : audit [INF] from='client.? 172.21.15.93:0/2646113948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a46f3336-3bc4-4736-9095-c4d72e662323"}]: dispatch 2021-11-20T08:54:26.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:54:26.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: 20T08:54:25.192078+0000 mon.smithi093 (mon.0) 233 : audit [INF] from='client.? 172.21.15.93:0/2646113948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a46f3336-3bc4-4736-9095-c4d72e662323"}]': finished 2021-11-20T08:54:26.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: cluster 2021-11-20T08:54:25.192121+0000 mon.smithi093 (mon.0) 234 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-20T08:54:26.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: audit 2021-11-20T08:54:25.192249+0000 2021-11-20T08:54:26.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: mon.smithi093 (mon.0) 235 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:26.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: 2021-11-20T08:54:26.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: cluster 2021-11-20T08:54 2021-11-20T08:54:26.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: :25.497416+0000 mgr.smithi093.qdikkm (mgr.14162) 74 : 2021-11-20T08:54:26.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:26.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:54:26.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: T08:54:25.597364+0000 mon.smithi093 (mon. 2021-11-20T08:54:26.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:25 smithi102 conmon[36472]: 0) 236 : audit [DBG] from='client.? 172.21.15.93:0/664788105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:54:27.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:27 smithi102 conmon[36472]: cluster 2021-11-20T08:54:27.497685+0000 mgr.smithi093.qdikkm (mgr.14162) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:27.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:27 smithi093 conmon[33522]: cluster 2021-11-20T08:54:27.497685+0000 mgr.smithi093.qdikkm (mgr.14162) 75 2021-11-20T08:54:27.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:27 smithi093 conmon[33522]: : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:29.903 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:29 smithi093 conmon[33522]: cluster 2021-11-20T08:54:29.497942+0000 mgr.smithi093.qdikkm (mgr.14162) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:29.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:29 smithi102 conmon[36472]: cluster 2021-11-20T08:54:29.497942+0000 mgr.smithi093.qdikkm (mgr.14162) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:32.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: cluster 2021-11-20T08:54:31.498383+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:54:32.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: ) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:32.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: audit 2021-11-20T08:54:32.409925+0000 mon.smithi093 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:32.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: audit 2021-11-20T08:54:32.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: 2021-11-20T08:54:32.410581+0000 mon.smithi093 (mon.0) 238 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T08:54:32.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:54:32.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: 11-20T08:54:32.411152+0000 mon.smithi093 (mon.0) 239 : 2021-11-20T08:54:32.667 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:32 smithi093 conmon[33522]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:32.838 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:32 smithi102 conmon[36472]: cluster 2021-11-20T08:54:31.498383+0000 mgr.smithi093.qdikkm (mgr.14162) 77 : cluster [DBG] 2021-11-20T08:54:32.838 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:32 smithi102 conmon[36472]: pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:32.839 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:32 smithi102 conmon[36472]: audit 2021-11-20T08:54:32.409925+0000 mon.smithi093 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:32.839 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:32 smithi102 conmon[36472]: audit 2021-11-20T08:54:32.410581+0000 mon.smithi093 (mon.0) 238 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T08:54:32.839 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:32 smithi102 conmon[36472]: audit 2021-11-20T08:54:32.411152+0000 mon.smithi093 (mon.0) 239 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:33.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:33 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:32.410383+0000 mgr.smithi093.qdikkm 2021-11-20T08:54:33.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:33 smithi093 conmon[33522]: (mgr.14162) 78 : cephadm [INF] Reconfiguring crash.smithi102 (monmap changed)... 2021-11-20T08:54:33.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:33 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:32.411543+0000 mgr.smithi093.qdikkm (mgr.14162) 79 : cephadm [INF] Reconfiguring daemon crash.smithi102 on smithi102 2021-11-20T08:54:33.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:33 smithi093 conmon[33522]: audit 2021-11-20T08:54:33.403501+0000 mon.smithi093 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T08:54:33.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:33 smithi093 conmon[33522]: audit 2021-11-20T08:54:33.404053+0000 mon.smithi093 (mon.0) 241 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:33.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:33 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:32.410383+0000 mgr.smithi093.qdikkm (mgr.14162) 78 : cephadm [INF] Reconfiguring crash.smithi102 (monmap changed)... 2021-11-20T08:54:33.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:33 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:32.411543+0000 mgr.smithi093.qdikkm (mgr.14162) 79 : cephadm [INF] Reconfiguring daemon crash.smithi102 on smithi102 2021-11-20T08:54:33.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:33 smithi102 conmon[36472]: audit 2021-11-20T08:54:33.403501+0000 mon.smithi093 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T08:54:33.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:33 smithi102 conmon[36472]: audit 2021-11-20T08:54:33.404053+0000 mon.smithi093 (mon.0) 241 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:34.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:34 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:33.404480+0000 mgr.smithi093.qdikkm (mgr.14162) 80 : cephadm [INF] Deploying daemon osd.1 on smithi093 2021-11-20T08:54:34.506 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:34 smithi102 conmon[36472]: cluster 2021-11-20T08:54:33.498634+0000 mgr.smithi093.qdikkm (mgr.14162) 81 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:34.506 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:34 smithi102 conmon[36472]: audit 2021-11-20T08:54:33.587983+0000 mon.smithi093 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:34.506 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:34 smithi102 conmon[36472]: audit 2021-11-20T08:54:33.588816+0000 mon.smithi093 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.ievjft", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T08:54:34.507 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:34 smithi102 conmon[36472]: audit 2021-11-20T08:54:33.589492+0000 mon.smithi093 (mon.0) 244 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T08:54:34.507 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:34 smithi102 conmon[36472]: audit 2021-11-20T08:54:33.590029+0000 mon.smithi093 (mon.0) 245 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:34.590 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:34 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:33.404480+0000 mgr.smithi093.qdikkm (mgr.14162) 80 : cephadm [INF] Deploying daemon osd.1 on smithi093 2021-11-20T08:54:34.590 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:34 smithi093 conmon[33522]: cluster 2021-11-20T08:54:33.498634+0000 mgr.smithi093.qdikkm (mgr.14162) 81 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:34.591 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:34 smithi093 conmon[33522]: audit 2021-11-20T08:54:33.587983+0000 mon.smithi093 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:34.591 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:34 smithi093 conmon[33522]: audit 2021-11-20T08:54:33.588816+0000 mon.smithi093 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.ievjft", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T08:54:34.592 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:34 smithi093 conmon[33522]: audit 2021-11-20T08:54:33.589492+0000 mon.smithi093 (mon.0) 244 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T08:54:34.592 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:34 smithi093 conmon[33522]: audit 2021-11-20T08:54:33.590029+0000 mon.smithi093 (mon.0) 245 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:35.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:33.588543+0000 mgr.smithi093.qdikkm (mgr.14162) 82 : cephadm 2021-11-20T08:54:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: [INF] Reconfiguring mgr.smithi102.ievjft (monmap changed)... 2021-11-20T08:54:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:33.590524+0000 mgr.smithi093.qdikkm (mgr.14162) 83 : cephadm [INF] Reconfiguring daemon mgr.smithi102.ievjft on smithi102 2021-11-20T08:54:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: audit 2021-11-20T08:54:34.754691+0000 mon.smithi093 (mon.0) 246 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: audit 2021-11-20T08:54:34.755427+0000 mon.smithi093 (mon.0) 247 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T08:54:35.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: audit 2021-11-20T08:54:34.756010+0000 mon.smithi093 ( 2021-11-20T08:54:35.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: mon.0) 248 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T08:54:35.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: audit 2021-11-20T08:54:34.756569+0000 2021-11-20T08:54:35.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:35 smithi102 conmon[36472]: mon.smithi093 (mon.0) 249 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:35.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:33.588543+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:35.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: 14162) 82 : cephadm [INF] Reconfiguring mgr.smithi102.ievjft (monmap changed)... 2021-11-20T08:54:35.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:33.590524+0000 mgr.smithi093.qdikkm (mgr.14162) 83 : cephadm [INF] Reconfiguring daemon mgr.smithi102.ievjft on smithi102 2021-11-20T08:54:35.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: audit 2021-11-20T08:54:35.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: 2021-11-20T08:54:34.754691+0000 mon.smithi093 (mon.0) 246 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:35.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:54:35.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: 20T08:54:34.755427+0000 mon.smithi093 ( 2021-11-20T08:54:35.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: mon.0) 247 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T08:54:35.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:54:35.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: 11-20T08:54:34.756010+0000 2021-11-20T08:54:35.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: mon.smithi093 (mon.0) 248 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T08:54:35.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: 2021-11-20T08:54:35.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: audit 2021-11-20T08:54:35.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: 2021-11-20T08:54:34.756569+0000 mon.smithi093 (mon 2021-11-20T08:54:35.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:35 smithi093 conmon[33522]: .0) 249 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:36.298 INFO:teuthology.orchestra.run.smithi093.stdout:Created osd(s) 1 on host 'smithi093' 2021-11-20T08:54:36.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.297+0000 7f7f0dffb700 1 -- 172.21.15.93:0/1152886602 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7f1c105510 con 0x7f7f0805c2b0 2021-11-20T08:54:36.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7f0805c2b0 msgr2=0x7f7f0805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:36.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7f0805c2b0 0x7f7f0805e770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7f1800b790 tx=0x7f7f18008040).stop 2021-11-20T08:54:36.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 msgr2=0x7f7f1c10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:36.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c10d5e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f7f10009f50 tx=0x7f7f10009b30).stop 2021-11-20T08:54:36.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 shutdown_connections 2021-11-20T08:54:36.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7f0805c2b0 0x7f7f0805e770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:36.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f1c060390 0x7f7f1c10d5e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:36.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 --2- 172.21.15.93:0/1152886602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c0f7640 0x7f7f1c10dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:36.303 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 >> 172.21.15.93:0/1152886602 conn(0x7f7f1c0f0de0 msgr2=0x7f7f1c0f31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:54:36.303 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 shutdown_connections 2021-11-20T08:54:36.304 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:36.300+0000 7f7f23406700 1 -- 172.21.15.93:0/1152886602 wait complete. 2021-11-20T08:54:36.592 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:34.755218 2021-11-20T08:54:36.592 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 84 : cephadm [INF] Reconfiguring mon.smithi102 (monmap changed)... 2021-11-20T08:54:36.593 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:34.757033+0000 mgr.smithi093.qdikkm (mgr.14162) 85 : cephadm [INF] Reconfiguring daemon mon.smithi102 on smithi102 2021-11-20T08:54:36.593 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: cluster 2021-11-20T08:54:35.498911+0000 mgr.smithi093.qdikkm (mgr.14162) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:36.593 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:35.918059+0000 mon.smithi093 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.593 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:35.921178+0000 mon.smithi093 (mon.0) 251 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T08:54:36.594 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.294377+0000 mon.smithi093 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.594 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.297870+0000 mon.smithi093 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.594 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.310931+0000 mon.smithi093 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.93:9093"}]: dispatch 2021-11-20T08:54:36.594 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.314959+0000 mon.smithi093 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.594 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.318222+0000 mon.smithi093 (mon.0) 256 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T08:54:36.595 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.323057+0000 mon.smithi093 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.93:3000"}]: dispatch 2021-11-20T08:54:36.595 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:54:36.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: 20T08:54:36.327185+0000 mon.smithi093 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.331553+0000 mon.smithi093 (mon.0) 259 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T08:54:36.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36 2021-11-20T08:54:36.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: .336719+0000 mon.smithi093 (mon.0) 260 : audit 2021-11-20T08:54:36.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.93:9095"}]: dispatch 2021-11-20T08:54:36.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:54:36.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: 54:36.341211+0000 mon.smithi093 (mon.0) 2021-11-20T08:54:36.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: 261 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.598 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: 2021-11-20T08:54:36 2021-11-20T08:54:36.598 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: .343782+0000 mon.smithi093 (mon.0) 262 : 2021-11-20T08:54:36.598 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[33522]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:36.644 DEBUG:teuthology.orchestra.run.smithi093:osd.1> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.1.service 2021-11-20T08:54:36.647 INFO:tasks.cephadm:Deploying osd.2 on smithi093 with /dev/vg_nvme/lv_2... 2021-11-20T08:54:36.648 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T08:54:36.674 INFO:journalctl@ceph.osd.1.smithi093.stdout:-- Logs begin at Sat 2021-11-20 08:41:08 UTC. -- 2021-11-20T08:54:36.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:34.755218+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:54:36.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: ) 84 : cephadm [INF] Reconfiguring mon.smithi102 (monmap changed)... 2021-11-20T08:54:36.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:34.757033+0000 mgr.smithi093.qdikkm (mgr.14162) 85 : cephadm [INF] Reconfiguring daemon mon.smithi102 on smithi102 2021-11-20T08:54:36.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: cluster 2021-11-20T08:54:35.498911+0000 mgr.smithi093.qdikkm (mgr.14162) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:36.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021-11-20T08:54:35.918059+0000 mon.smithi093 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021-11-20T08:54:35.921178+0000 mon.smithi093 (mon.0) 251 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T08:54:36.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021-11-20T08:54:36.294377+0000 mon.smithi093 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 2021-11-20T08:54:36.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:54:36.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 20T08:54:36.297870 2021-11-20T08:54:36.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 253 : 2021-11-20T08:54:36.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021-11 2021-11-20T08:54:36.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: -20T08:54:36.310931+0000 2021-11-20T08:54:36.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: mon.smithi093 (mon.0) 254 : 2021-11-20T08:54:36.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.93:9093"}]: dispatch 2021-11-20T08:54:36.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:36.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: -11- 2021-11-20T08:54:36.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 20T08:54:36.314959 2021-11-20T08:54:36.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 255 2021-11-20T08:54:36.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:54:36.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 11-20T08:54:36.318222 2021-11-20T08:54:36.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 256 2021-11-20T08:54:36.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T08:54:36.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:36.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: -11-20T08:54:36. 2021-11-20T08:54:36.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 323057+0000 mon.smithi093 (mon.0) 2021-11-20T08:54:36.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 257 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.93:3000"}]: dispatch 2021-11-20T08:54:36.928 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:36.928 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: -11-20T08:54:36. 2021-11-20T08:54:36.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 327185+0000 mon.smithi093 (mon.0) 2021-11-20T08:54:36.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 258 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.930 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:36.930 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: -11-20T08:54:36.331553 2021-11-20T08:54:36.931 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 259 2021-11-20T08:54:36.931 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T08:54:36.932 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:36.932 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: -11-20T08:54:36. 2021-11-20T08:54:36.933 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 336719+0000 mon.smithi093 (mon.0) 2021-11-20T08:54:36.933 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 260 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.93:9095"}]: dispatch 2021-11-20T08:54:36.934 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:36.935 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: -11-20T08:54:36 2021-11-20T08:54:36.935 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: .341211+0000 mon.smithi093 (mon.0 2021-11-20T08:54:36.935 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: ) 261 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:36.936 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: audit 2021-11-20T08:54:36.936 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: 2021-11-20T08:54 2021-11-20T08:54:36.937 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: :36.343782+0000 mon.smithi093 ( 2021-11-20T08:54:36.937 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: mon.0) 262 : audit [DBG] 2021-11-20T08:54:36.937 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:36 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:36.962 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 08:54:36 smithi093 conmon[55484]: debug 2021-11-20T08:54:36.765+0000 7f266e05b080 -1 Falling back to public interface 2021-11-20T08:54:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:37 smithi093 conmon[33522]: audit 2021-11-20T08:54:35. 2021-11-20T08:54:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:37 smithi093 conmon[33522]: 921489+0000 mgr.smithi093.qdikkm (mgr.14162) 87 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T08:54:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:37 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.311245+0000 mgr.smithi093.qdikkm (mgr.14162) 88 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.93:9093"}]: dispatch 2021-11-20T08:54:37.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:37 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.318521+0000 mgr.smithi093.qdikkm (mgr.14162) 89 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T08:54:37.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:37 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.323311+0000 mgr.smithi093.qdikkm (mgr.14162) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.93:3000"}]: dispatch 2021-11-20T08:54:37.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:37 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.331910+0000 mgr.smithi093.qdikkm (mgr.14162) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T08:54:37.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:37 smithi093 conmon[33522]: audit 2021-11-20T08:54:36.337241+0000 mgr.smithi093.qdikkm (mgr.14162) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.93:9095"}]: dispatch 2021-11-20T08:54:37.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:37 smithi102 conmon[36472]: audit 2021-11-20T08:54:35.921489+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:54:37.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:37 smithi102 conmon[36472]: mgr.14162) 87 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T08:54:37.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:37 smithi102 conmon[36472]: audit 2021-11-20T08:54:36.311245+0000 mgr.smithi093.qdikkm (mgr.14162) 88 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.93:9093"}]: dispatch 2021-11-20T08:54:37.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:37 smithi102 conmon[36472]: audit 2021-11-20T08:54:36.318521+0000 mgr.smithi093.qdikkm (mgr.14162) 89 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T08:54:37.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:37 smithi102 conmon[36472]: audit 2021-11-20T08:54:36.323311+0000 mgr.smithi093.qdikkm (mgr.14162) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.93:3000"}]: dispatch 2021-11-20T08:54:37.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:37 smithi102 conmon[36472]: audit 2021-11-20T08:54:36.331910+0000 mgr.smithi093.qdikkm (mgr.14162) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T08:54:37.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:37 smithi102 conmon[36472]: audit 2021-11-20T08:54:36.337241+0000 mgr.smithi093.qdikkm (mgr.14162) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.93:9095"}]: dispatch 2021-11-20T08:54:38.492 INFO:teuthology.orchestra.run.smithi093.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T08:54:38.492 INFO:teuthology.orchestra.run.smithi093.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T08:54:38.492 INFO:teuthology.orchestra.run.smithi093.stderr: stderr: 10+0 records in 2021-11-20T08:54:38.493 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records out 2021-11-20T08:54:38.493 INFO:teuthology.orchestra.run.smithi093.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0153786 s, 682 MB/s 2021-11-20T08:54:38.493 INFO:teuthology.orchestra.run.smithi093.stderr:--> Zapping successful for: 2021-11-20T08:54:38.918 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:38 smithi093 conmon[33522]: cluster 2021-11-20T08:54:37.499154+0000 mgr.smithi093.qdikkm (mgr.14162) 93 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:38.918 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:38 smithi093 conmon[33522]: audit 2021-11-20T08:54:38.379322+0000 mon.smithi093 (mon.0) 263 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:38.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:38 smithi102 conmon[36472]: cluster 2021-11-20T08:54:37.499154+0000 mgr.smithi093.qdikkm (mgr.14162) 93 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:38.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:38 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:38.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:38 smithi102 conmon[36472]: -11-20T08:54:38.379322+0000 mon.smithi093 (mon.0) 263 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:38.937 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi093:vg_nvme/lv_2 2021-11-20T08:54:40.471 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 08:54:40 smithi093 conmon[55484]: debug 2021-11-20T08:54:40.286+0000 7f266e05b080 -1 osd.1 0 log_to_monitors {default=true} 2021-11-20T08:54:40.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 -- 172.21.15.93:0/1731373226 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f4d60 msgr2=0x7f37780f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:40.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 --2- 172.21.15.93:0/1731373226 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f4d60 0x7f37780f5180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f3774004660 tx=0x7f3774009b30).stop 2021-11-20T08:54:40.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 -- 172.21.15.93:0/1731373226 shutdown_connections 2021-11-20T08:54:40.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 --2- 172.21.15.93:0/1731373226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37780f5f60 0x7f37780f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:54:40.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 --2- 172.21.15.93:0/1731373226 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f4d60 0x7f37780f5180 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:40.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 -- 172.21.15.93:0/1731373226 >> 172.21.15.93:0/1731373226 conn(0x7f37780f04c0 msgr2=0x7f37780f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:54:40.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 -- 172.21.15.93:0/1731373226 shutdown_connections 2021-11-20T08:54:40.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.690+0000 7f3780242700 1 -- 172.21.15.93:0/1731373226 wait complete. 2021-11-20T08:54:40.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f3780242700 1 Processor -- start 2021-11-20T08:54:40.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f3780242700 1 -- start start 2021-11-20T08:54:40.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f3780242700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37780f4d60 0x7f3778109c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:40.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f3780242700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f5f60 0x7f377810c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:40.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f3780242700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37780fecc0 con 0x7f37780f5f60 2021-11-20T08:54:40.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f3780242700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37780fee00 con 0x7f37780f4d60 2021-11-20T08:54:40.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377d7dd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f5f60 0x7f377810c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:40.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377d7dd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f5f60 0x7f377810c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42620/0 (socket says 172.21.15.93:42620) 2021-11-20T08:54:40.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377d7dd700 1 -- 172.21.15.93:0/1488441479 learned_addr learned my addr 172.21.15.93:0/1488441479 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:54:40.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377d7dd700 1 -- 172.21.15.93:0/1488441479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37780f4d60 msgr2=0x7f3778109c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:40.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377dfde700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37780f4d60 0x7f3778109c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:40.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377d7dd700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37780f4d60 0x7f3778109c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:40.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377d7dd700 1 -- 172.21.15.93:0/1488441479 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3774005040 con 0x7f37780f5f60 2021-11-20T08:54:40.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377dfde700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37780f4d60 0x7f3778109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T08:54:40.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.691+0000 7f377d7dd700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f5f60 0x7f377810c140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f376800b410 tx=0x7f3768006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:40.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.692+0000 7f376e7fc700 1 -- 172.21.15.93:0/1488441479 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37680125f0 con 0x7f37780f5f60 2021-11-20T08:54:40.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.692+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377810c7a0 con 0x7f37780f5f60 2021-11-20T08:54:40.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.692+0000 7f376e7fc700 1 -- 172.21.15.93:0/1488441479 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3768012c00 con 0x7f37780f5f60 2021-11-20T08:54:40.698 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.692+0000 7f376e7fc700 1 -- 172.21.15.93:0/1488441479 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3768002aa0 con 0x7f37780f5f60 2021-11-20T08:54:40.698 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.692+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377810cd30 con 0x7f37780f5f60 2021-11-20T08:54:40.703 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.693+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37780f4150 con 0x7f37780f5f60 2021-11-20T08:54:40.704 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.693+0000 7f376e7fc700 1 -- 172.21.15.93:0/1488441479 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f376800f030 con 0x7f37780f5f60 2021-11-20T08:54:40.704 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.693+0000 7f376e7fc700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f376405c2b0 0x7f376405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:40.704 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.693+0000 7f376e7fc700 1 -- 172.21.15.93:0/1488441479 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2018+0+0 (secure 0 0 0) 0x7f3768075680 con 0x7f37780f5f60 2021-11-20T08:54:40.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.693+0000 7f377dfde700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f376405c2b0 0x7f376405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:40.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.694+0000 7f377dfde700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f376405c2b0 0x7f376405e770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3774009d60 tx=0x7f3774009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:40.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.697+0000 7f376e7fc700 1 -- 172.21.15.93:0/1488441479 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3768049000 con 0x7f37780f5f60 2021-11-20T08:54:40.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:40.887+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f37780fa5e0 con 0x7f376405c2b0 2021-11-20T08:54:40.887 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:40 smithi093 conmon[33522]: cluster 2021-11-20T08:54:39.499424+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:40.888 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:40 smithi093 conmon[33522]: 14162) 94 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:40.888 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:40 smithi093 conmon[33522]: audit 2021-11-20T08:54:40.290138+0000 mon.smithi093 (mon.0) 264 : audit [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T08:54:40.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:40 smithi102 conmon[36472]: cluster 2021-11-20T08:54:39.499424+0000 mgr.smithi093.qdikkm (mgr.14162) 94 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:40.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:40 smithi102 conmon[36472]: audit 2021-11-20T08:54:40.290138+0000 2021-11-20T08:54:40.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:40 smithi102 conmon[36472]: mon.smithi093 (mon.0) 264 : audit [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T08:54:41.711 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[55484]: debug 2021-11-20T08:54:41.512+0000 7f26645cc700 -1 osd.1 0 waiting for initial osdmap 2021-11-20T08:54:41.711 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[55484]: debug 2021-11-20T08:54:41.518+0000 7f265f545700 -1 osd.1 13 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T08:54:41.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021-11-20T08:54:40 2021-11-20T08:54:41.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: .512425+0000 mon.smithi093 (mon.0) 265 : audit [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T08:54:41.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: cluster 2021-11-20T08:54:41.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 2021-11-20T08:54:40.512488+0000 mon.smithi093 (mon.0) 2021-11-20T08:54:41.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 266 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T08:54:41.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:54:41.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: -20T08:54:40.513258+0000 mon.smithi093 ( 2021-11-20T08:54:41.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: mon.0) 267 : 2021-11-20T08:54:41.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021-11-20T08:54:41.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:41.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:54:41.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 11-20T08:54:40.513578+0000 2021-11-20T08:54:41.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: mon.smithi093 (mon.0) 268 : audit 2021-11-20T08:54:41.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]: dispatch 2021-11-20T08:54:41.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:54:41.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: -20T08:54:40.889482+0000 mgr.smithi093.qdikkm 2021-11-20T08:54:41.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: (mgr.14162) 95 : audit 2021-11-20T08:54:41.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: [DBG] from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:41.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021-11-20T08:54:41.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 2021-11-20T 2021-11-20T08:54:41.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 08:54:40.890222+0000 mon.smithi093 2021-11-20T08:54:41.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: (mon.0) 269 : audit 2021-11-20T08:54:41.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:41.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: cephadm 2021-11-20 2021-11-20T08:54:41.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: T08:54:40.891263+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:54:41.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: .14162) 96 : cephadm [INF] Applying service osd.None on host smithi093... 2021-11-20T08:54:41.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 2021-11-20T08:54:41.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:54:41.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 11-20T08:54:40.891443+0000 mon.smithi093 ( 2021-11-20T08:54:41.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: mon.0) 270 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:41.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:54:41.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: -20T08:54:40.891930+0000 mon.smithi093 (mon 2021-11-20T08:54:41.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: .0) 271 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:41.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 2021-11-20T08:54:41.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: cluster 2021-11-20T08:54:41.721 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 2021-11-20T08:54: 2021-11-20T08:54:41.721 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 41.499779+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:41.721 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 14162) 97 : cluster [DBG] 2021-11-20T08:54:41.721 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:41.721 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021-11-20T08:54:41.722 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 2021-11-20T08:54:41.510460+0000 2021-11-20T08:54:41.722 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: mon.smithi093 (mon.0) 272 : 2021-11-20T08:54:41.722 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]': finished 2021-11-20T08:54:41.722 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: cluster 2021-11 2021-11-20T08:54:41.723 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: -20T08:54:41.510512+0000 2021-11-20T08:54:41.723 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: mon.smithi093 (mon.0) 273 : cluster 2021-11-20T08:54:41.723 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T08:54:41.723 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:54:41.724 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 11-20T08:54:41.510634+0000 2021-11-20T08:54:41.724 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: mon.smithi093 (mon.0) 274 2021-11-20T08:54:41.724 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:41.724 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: audit 2021 2021-11-20T08:54:41.724 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: -11-20T08:54:41.513203 2021-11-20T08:54:41.725 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 2021-11-20T08:54:41.725 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:41 smithi093 conmon[33522]: 275 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:41.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021-11-20T08:54:40.512425+0000 mon.smithi093 ( 2021-11-20T08:54:41.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: mon.0) 265 : audit [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T08:54:41.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: cluster 2021-11-20T08:54:40.512488+0000 mon.smithi093 (mon.0) 266 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T08:54:41.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021-11-20T08:54:40.513258+0000 mon.smithi093 (mon.0) 267 : 2021-11-20T08:54:41.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:41.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021-11-20T08: 2021-11-20T08:54:41.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 54:40.513578+0000 mon.smithi093 (mon.0) 268 : 2021-11-20T08:54:41.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]: dispatch 2021-11-20T08:54:41.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021-11-20T08 2021-11-20T08:54:41.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: :54:40.889482+0000 mgr.smithi093.qdikkm (mgr.14162) 95 2021-11-20T08:54:41.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: : audit [DBG] from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi093:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:41.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021-11-20T08:54 2021-11-20T08:54:41.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: :40.890222+0000 mon.smithi093 (mon.0) 269 : 2021-11-20T08:54:41.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:41.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: cephadm 2021-11-20T08: 2021-11-20T08:54:41.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 54:40.891263+0000 mgr.smithi093.qdikkm (mgr.14162) 96 2021-11-20T08:54:41.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: : cephadm [INF] Applying service osd.None on host smithi093... 2021-11-20T08:54:41.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021-11-20T08: 2021-11-20T08:54:41.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 54:40.891443+0000 mon.smithi093 (mon.0) 270 : 2021-11-20T08:54:41.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:41.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021-11-20T08:54:40. 2021-11-20T08:54:41.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 891930+0000 mon.smithi093 (mon.0) 271 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:41.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 2021-11-20T08:54:41.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:54:41.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: T08:54:41.499779+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:41.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 14162) 97 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:41.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:54:41.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 11-20T08:54:41.510460+0000 mon.smithi093 2021-11-20T08:54:41.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: (mon.0) 272 : audit [INF] from='osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]': finished 2021-11-20T08:54:41.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:54:41.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: -11-20T08:54:41.510512+0000 mon.smithi093 2021-11-20T08:54:41.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: (mon.0) 273 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T08:54:41.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:54:41.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: 11-20T08:54:41.510634+0000 2021-11-20T08:54:41.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: mon.smithi093 (mon.0) 274 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:41.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: audit 2021 2021-11-20T08:54:41.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: -11-20T08:54:41.513203+0000 2021-11-20T08:54:41.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:41 smithi102 conmon[36472]: mon.smithi093 (mon.0) 275 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:42.943 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:42 smithi093 conmon[33522]: audit 2021-11-20T08:54:41.815970+0000 mon.smithi093 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:42.943 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:42 smithi093 conmon[33522]: audit 2021-11-20T08:54:42.513062+0000 mon.smithi093 (mon.0) 277 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:42.943 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:42 smithi093 conmon[33522]: cluster 2021-11-20T08:54:42.522335+0000 mon.smithi093 (mon.0) 278 : cluster [INF] osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] boot 2021-11-20T08:54:42.944 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:42 smithi093 conmon[33522]: cluster 2021-11-20T08:54:42.522372+0000 mon.smithi093 (mon.0) 279 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T08:54:42.944 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:42 smithi093 conmon[33522]: audit 2021-11-20T08:54:42.522623+0000 mon.smithi093 (mon.0) 280 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:43.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: audit 2021-11-20T08:54:41.815970+0000 mon.smithi093 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:43.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: audit 2021-11-20T08:54:42.513062+0000 mon.smithi093 (mon.0) 277 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:43.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: cluster 2021-11-20T08:54:42.522335 2021-11-20T08:54:43.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 278 : cluster [INF] osd.1 [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] boot 2021-11-20T08:54:43.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: cluster 2021-11-20T08:54:42 2021-11-20T08:54:43.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: .522372+0000 mon.smithi093 (mon.0) 279 : cluster 2021-11-20T08:54:43.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T08:54:43.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: audit 2021-11-20T08:54 2021-11-20T08:54:43.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: :42.522623+0000 mon.smithi093 (mon.0) 280 2021-11-20T08:54:43.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:42 smithi102 conmon[36472]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T08:54:44.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: cluster 2021-11-20T08:54:41.314240+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:54:44.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: cluster 2021-11-20T08:54:41.314336+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:54:44.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: audit 2021-11-20T08:54:43.051170+0000 2021-11-20T08:54:44.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: mon.smithi093 (mon.0) 281 : audit [INF] from='client.? 172.21.15.93:0/310184768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72096543-af61-429d-a8d1-a77726a1f3eb"}]: dispatch 2021-11-20T08:54:44.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: audit 2021-11 2021-11-20T08:54:44.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: -20T08:54:43.056529+0000 mon.smithi093 (mon 2021-11-20T08:54:44.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: .0) 282 : audit [INF] from='client.? 172.21.15.93:0/310184768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72096543-af61-429d-a8d1-a77726a1f3eb"}]': finished 2021-11-20T08:54:44.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T08:54:44.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: 20T08:54:43.056579+0000 mon.smithi093 (mon. 2021-11-20T08:54:44.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: 0) 283 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-20T08:54:44.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: audit 2021-11 2021-11-20T08:54:44.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: -20T08:54:43.056677+0000 mon.smithi093 ( 2021-11-20T08:54:44.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: mon.0) 284 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:44.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:54:44.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: T08:54:43.465109+0000 mon.smithi093 (mon 2021-11-20T08:54:44.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: .0) 285 : audit [DBG] from='client.? 172.21.15.93:0/3062945108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:54:44.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: cluster 2021-11-20T08:54:43 2021-11-20T08:54:44.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: .500060+0000 mgr.smithi093.qdikkm (mgr.14162) 98 : cluster [DBG] 2021-11-20T08:54:44.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:43 smithi102 conmon[36472]: pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:44.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: cluster 2021-11-20T08:54:41.314240+0000 2021-11-20T08:54:44.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:54:44.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: cluster 2021-11-20T08:54:41.314336+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:54:44.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: audit 2021-11-20T08:54:43.051170+0000 mon.smithi093 (mon.0) 281 : audit [INF] from='client.? 172.21.15.93:0/310184768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72096543-af61-429d-a8d1-a77726a1f3eb"}]: dispatch 2021-11-20T08:54:44.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: audit 2021-11-20T08:54:43.056529+0000 mon.smithi093 (mon.0) 282 : audit [INF] from='client.? 172.21.15.93:0/310184768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72096543-af61-429d-a8d1-a77726a1f3eb"}]': finished 2021-11-20T08:54:44.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: cluster 2021-11-20T08:54:43.056579+0000 mon.smithi093 (mon.0) 283 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-20T08:54:44.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: audit 2021-11-20T08:54:43.056677+0000 mon.smithi093 (mon.0) 284 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:44.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: audit 2021-11-20T08:54:43.465109+0000 mon.smithi093 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.93:0/3062945108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:54:44.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:43 smithi093 conmon[33522]: cluster 2021-11-20T08:54:43.500060+0000 mgr.smithi093.qdikkm (mgr.14162) 98 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T08:54:45.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:45 smithi102 conmon[36472]: cluster 2021-11-20T08:54:44.063461+0000 mon.smithi093 (mon.0) 286 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T08:54:45.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:45 smithi102 conmon[36472]: audit 2021-11-20T08:54:44.063563+0000 mon.smithi093 (mon.0) 287 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:45.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:45 smithi093 conmon[33522]: cluster 2021-11-20T08:54:44.063461+0000 mon.smithi093 (mon.0) 286 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T08:54:45.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:45 smithi093 conmon[33522]: audit 2021-11-20T08:54:44.063563+0000 mon.smithi093 (mon.0) 287 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:46.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:46 smithi102 conmon[36472]: cluster 2021-11-20T08:54:45.500343+0000 mgr.smithi093.qdikkm (mgr.14162) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:46 smithi093 conmon[33522]: cluster 2021-11-20T08:54:45.500343+0000 mgr.smithi093.qdikkm (mgr.14162) 99 : 2021-11-20T08:54:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:46 smithi093 conmon[33522]: cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:47.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:47 smithi102 conmon[36472]: cluster 2021-11-20T08:54:47.500559+0000 mgr.smithi093.qdikkm (mgr.14162) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:47.924 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:47 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:54:47.924 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:47 smithi093 conmon[33522]: 54:47.500559+0000 mgr.smithi093.qdikkm (mgr.14162) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:50.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:48.901904+0000 mgr.smithi093.qdikkm (mgr.14162) 101 : cephadm [INF] Detected new or changed devices on smithi093 2021-11-20T08:54:50.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: audit 2021-11-20T08:54:48. 2021-11-20T08:54:50.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: 907624+0000 mon.smithi093 (mon.0) 288 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:50.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: audit 2021-11-20T08:54:48.912928+0000 mon.smithi093 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:50.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:54:50.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: 20T08:54:48.915600+0000 mon.smithi093 (mon.0) 290 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:50.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T08:54:50.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: 08:54:49.500960+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:50.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: 14162) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:50.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:54:50.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: 20T08:54:49.625449+0000 mon.smithi093 ( 2021-11-20T08:54:50.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T08:54:50.175 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: audit 2021-11 2021-11-20T08:54:50.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: -20T08:54:49.626046+0000 mon.smithi093 2021-11-20T08:54:50.176 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:49 smithi102 conmon[36472]: (mon.0) 292 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:50.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:48.901904 2021-11-20T08:54:50.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 101 : cephadm [INF] Detected new or changed devices on smithi093 2021-11-20T08:54:50.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: audit 2021-11-20T08:54:48.907624+0000 mon.smithi093 (mon.0) 288 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:50.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: audit 2021-11-20T08:54:48.912928+0000 mon.smithi093 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:50.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: audit 2021-11-20T08:54:48.915600+0000 mon.smithi093 (mon.0) 290 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:50.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: cluster 2021-11-20T08:54:49.500960+0000 mgr.smithi093.qdikkm (mgr.14162) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:50.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: audit 2021-11-20T08:54:49.625449+0000 mon.smithi093 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T08:54:50.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:49 smithi093 conmon[33522]: audit 2021-11-20T08:54:49.626046+0000 mon.smithi093 (mon.0) 292 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:51.085 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:50 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:49.626534+0000 mgr.smithi093.qdikkm (mgr.14162) 103 : cephadm [INF] Deploying daemon osd.2 on smithi093 2021-11-20T08:54:51.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:50 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:49.626534+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:54:51.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:50 smithi102 conmon[36472]: 103 : cephadm [INF] Deploying daemon osd.2 on smithi093 2021-11-20T08:54:52.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:52 smithi102 conmon[36472]: audit 2021-11-20T08:54:51.054998+0000 mon.smithi093 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:52.423 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:52 smithi102 conmon[36472]: 2021-11-20T08:54:52.423 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:52 smithi102 conmon[36472]: cluster 2021-11-20T08:54:51.501404+0000 mgr.smithi093.qdikkm (mgr.14162) 104 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:52.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:52 smithi093 conmon[33522]: audit 2021-11-20T08:54:51.054998+0000 mon.smithi093 ( 2021-11-20T08:54:52.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:52 smithi093 conmon[33522]: mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:52.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:52 smithi093 conmon[33522]: cluster 2021-11-20T08:54:51.501404+0000 mgr.smithi093.qdikkm (mgr.14162) 104 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:52.920 INFO:teuthology.orchestra.run.smithi093.stdout:Created osd(s) 2 on host 'smithi093' 2021-11-20T08:54:52.920 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.919+0000 7f376e7fc700 1 -- 172.21.15.93:0/1488441479 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f37780fa5e0 con 0x7f376405c2b0 2021-11-20T08:54:52.922 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.922+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f376405c2b0 msgr2=0x7f376405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:52.922 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.922+0000 7f3780242700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f376405c2b0 0x7f376405e770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3774009d60 tx=0x7f3774009b30).stop 2021-11-20T08:54:52.922 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.922+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f5f60 msgr2=0x7f377810c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:52.923 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.922+0000 7f3780242700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f5f60 0x7f377810c140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f376800b410 tx=0x7f3768006ef0).stop 2021-11-20T08:54:52.925 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.926+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 shutdown_connections 2021-11-20T08:54:52.926 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.926+0000 7f3780242700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f376405c2b0 0x7f376405e770 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:52.926 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.926+0000 7f3780242700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37780f4d60 0x7f3778109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:52.926 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.926+0000 7f3780242700 1 --2- 172.21.15.93:0/1488441479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37780f5f60 0x7f377810c140 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:52.926 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.926+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 >> 172.21.15.93:0/1488441479 conn(0x7f37780f04c0 msgr2=0x7f37780f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:54:52.927 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.926+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 shutdown_connections 2021-11-20T08:54:52.927 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:54:52.926+0000 7f3780242700 1 -- 172.21.15.93:0/1488441479 wait complete. 2021-11-20T08:54:53.275 DEBUG:teuthology.orchestra.run.smithi093:osd.2> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.2.service 2021-11-20T08:54:53.279 INFO:tasks.cephadm:Deploying osd.3 on smithi102 with /dev/vg_nvme/lv_4... 2021-11-20T08:54:53.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T08:54:53.309 INFO:journalctl@ceph.osd.2.smithi093.stdout:-- Logs begin at Sat 2021-11-20 08:41:08 UTC. -- 2021-11-20T08:54:53.707 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 08:54:53 smithi093 conmon[61773]: debug 2021-11-20T08:54:53.351+0000 7f55fa07b080 -1 Falling back to public interface 2021-11-20T08:54:53.956 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:53 smithi102 conmon[36472]: audit 2021-11-20T08:54:52.914650+0000 mon.smithi093 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:53.956 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:53 smithi102 conmon[36472]: 2021-11-20T08:54:53.956 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:53 smithi102 conmon[36472]: audit 2021-11-20T08:54:52.920685+0000 mon.smithi093 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:53.956 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:53 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T08:54:53.957 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:53 smithi102 conmon[36472]: 20T08:54:53.501899+0000 mgr.smithi093.qdikkm (mgr.14162) 105 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:53.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:53 smithi093 conmon[33522]: audit 2021-11-20T08:54:52.914650+0000 mon.smithi093 ( 2021-11-20T08:54:53.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:53 smithi093 conmon[33522]: mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:53.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:53 smithi093 conmon[33522]: audit 2021-11-20T08:54:52.920685+0000 mon.smithi093 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:53.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:53 smithi093 conmon[33522]: cluster 2021-11-20T08:54:53.501899+0000 mgr.smithi093.qdikkm (mgr.14162) 105 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T08:54:55.110 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T08:54:55.110 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2021-11-20T08:54:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2021-11-20T08:54:55.111 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108693 s, 965 MB/s 2021-11-20T08:54:55.111 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2021-11-20T08:54:55.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:55 smithi102 conmon[36472]: audit 2021-11-20T08:54:54.311153+0000 mon.smithi093 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:55.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2021-11-20T08:54:55.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:55 smithi093 conmon[33522]: audit 2021-11-20T08:54:54.311153+0000 mon.smithi093 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:56.615 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:56 smithi093 conmon[33522]: cluster 2021-11-20T08:54:55. 2021-11-20T08:54:56.616 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:56 smithi093 conmon[33522]: 502161+0000 mgr.smithi093.qdikkm (mgr.14162) 106 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:56.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:56 smithi102 conmon[36472]: cluster 2021-11-20T08:54:55.502161+0000 mgr.smithi093.qdikkm (mgr.14162) 106 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:57.105 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 -- 172.21.15.102:0/145085813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 msgr2=0x7f7e680f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:57.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/145085813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e680f6b40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f7e58004660 tx=0x7f7e58009b30).stop 2021-11-20T08:54:57.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 -- 172.21.15.102:0/145085813 shutdown_connections 2021-11-20T08:54:57.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/145085813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e680f6b40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:57.106 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/145085813 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e680f5440 0x7f7e680f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:57.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 -- 172.21.15.102:0/145085813 >> 172.21.15.102:0/145085813 conn(0x7f7e680f0ba0 msgr2=0x7f7e680f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:54:57.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 -- 172.21.15.102:0/145085813 shutdown_connections 2021-11-20T08:54:57.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.101+0000 7f7e6e05b700 1 -- 172.21.15.102:0/145085813 wait complete. 2021-11-20T08:54:57.107 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6e05b700 1 Processor -- start 2021-11-20T08:54:57.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6e05b700 1 -- start start 2021-11-20T08:54:57.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6e05b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e680f5440 0x7f7e68109a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:57.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6e05b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e6810bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:57.108 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6e05b700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e68060420 con 0x7f7e680f5440 2021-11-20T08:54:57.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6e05b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e68060560 con 0x7f7e680f66e0 2021-11-20T08:54:57.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6c858700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e6810bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:57.109 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6c858700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e6810bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53704/0 (socket says 172.21.15.102:53704) 2021-11-20T08:54:57.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6c858700 1 -- 172.21.15.102:0/1550886973 learned_addr learned my addr 172.21.15.102:0/1550886973 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:54:57.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6d059700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e680f5440 0x7f7e68109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:57.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6c858700 1 -- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e680f5440 msgr2=0x7f7e68109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:54:57.110 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6c858700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e680f5440 0x7f7e68109a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:54:57.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6c858700 1 -- 172.21.15.102:0/1550886973 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58005040 con 0x7f7e680f66e0 2021-11-20T08:54:57.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.102+0000 7f7e6c858700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e6810bf70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f7e58009f50 tx=0x7f7e58009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:57.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.103+0000 7f7e5dffb700 1 -- 172.21.15.102:0/1550886973 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e58018070 con 0x7f7e680f66e0 2021-11-20T08:54:57.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.103+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6810c520 con 0x7f7e680f66e0 2021-11-20T08:54:57.111 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.103+0000 7f7e5dffb700 1 -- 172.21.15.102:0/1550886973 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e58013410 con 0x7f7e680f66e0 2021-11-20T08:54:57.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.103+0000 7f7e5dffb700 1 -- 172.21.15.102:0/1550886973 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e5801c3c0 con 0x7f7e680f66e0 2021-11-20T08:54:57.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.103+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6810ca50 con 0x7f7e680f66e0 2021-11-20T08:54:57.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.104+0000 7f7e5dffb700 1 -- 172.21.15.102:0/1550886973 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7e5801d420 con 0x7f7e680f66e0 2021-11-20T08:54:57.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.104+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e68059100 con 0x7f7e680f66e0 2021-11-20T08:54:57.112 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.105+0000 7f7e5dffb700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e5405c300 0x7f7e5405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:54:57.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.105+0000 7f7e5dffb700 1 -- 172.21.15.102:0/1550886973 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2429+0+0 (secure 0 0 0) 0x7f7e58077070 con 0x7f7e680f66e0 2021-11-20T08:54:57.113 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.105+0000 7f7e6d059700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e5405c300 0x7f7e5405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:54:57.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.105+0000 7f7e6d059700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e5405c300 0x7f7e5405e7c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f7e680600a0 tx=0x7f7e6400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:54:57.114 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.109+0000 7f7e5dffb700 1 -- 172.21.15.102:0/1550886973 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e5804a800 con 0x7f7e680f66e0 2021-11-20T08:54:57.211 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 08:54:56 smithi093 conmon[61773]: debug 2021-11-20T08:54:56.921+0000 7f55fa07b080 -1 osd.2 0 log_to_monitors {default=true} 2021-11-20T08:54:57.291 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:54:57.292+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e680008d0 con 0x7f7e5405c300 2021-11-20T08:54:57.781 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:56.617799+0000 mon.smithi093 (mon. 2021-11-20T08:54:57.782 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: 0) 297 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:54:57.782 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:56.617996+0000 mon.smithi093 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:54:57.782 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:56.925794+0000 mon.smithi093 (mon.0) 299 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T08:54:57.782 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.108672+0000 mon.smithi093 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:57.783 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.114199+0000 mon.smithi093 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:57.783 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.116344+0000 mon.smithi093 (mon.0) 302 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:57.783 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.295130+0000 mon.smithi093 (mon.0) 303 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:57.783 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.296628+0000 mon.smithi093 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:57.783 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:57 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.297419+0000 mon.smithi093 (mon.0) 305 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:56.617799+0000 mon.smithi093 ( 2021-11-20T08:54:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:54:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:56.617996+0000 mon.smithi093 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:54:57.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:56.925794+0000 mon.smithi093 (mon.0) 299 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T08:54:57.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:57.108672+0000 mon.smithi093 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:57.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:57.114199+0000 mon.smithi093 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:54:57.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:57.116344+0000 mon.smithi093 (mon.0) 302 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:54:57.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:57.295130+0000 mon.smithi093 (mon.0) 303 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:54:57.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:57.296628+0000 mon.smithi093 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:54:57.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:57 smithi093 conmon[33522]: audit 2021-11-20T08:54:57.297419+0000 mon.smithi093 (mon.0) 305 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:54:58.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.294376+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:54:58.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: 14162) 107 : audit [DBG] from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:58.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: cephadm 2021-11-20T08:54:57.296392+0000 mgr.smithi093.qdikkm (mgr.14162) 108 : cephadm [INF] Applying service osd.None on host smithi102... 2021-11-20T08:54:58.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.490574+0000 mon.smithi093 (mon.0) 306 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T08:54:58.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: cluster 2021-11-20T08:54:57.490634+0000 mon.smithi093 (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T08:54:58.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.490937+0000 mon.smithi093 (mon.0) 308 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:58.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: audit 2021-11-20T08:54:57.491188+0000 mon.smithi093 (mon.0) 309 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]: dispatch 2021-11-20T08:54:58.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:58 smithi102 conmon[36472]: cluster 2021-11-20T08:54:57.502519+0000 mgr.smithi093.qdikkm (mgr.14162) 109 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:58.961 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[61773]: debug 2021-11-20T08:54:58.494+0000 7f55f05ec700 -1 osd.2 0 waiting for initial osdmap 2021-11-20T08:54:58.961 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[61773]: debug 2021-11-20T08:54:58.499+0000 7f55ead64700 -1 osd.2 18 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T08:54:58.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:54:58.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 54:57.294376+0000 mgr.smithi093.qdikkm (mgr.14162) 107 : audit [DBG] from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:54:58.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: cephadm 2021-11-20T08:54:57.296392+0000 mgr.smithi093.qdikkm (mgr.14162) 108 : cephadm 2021-11-20T08:54:58.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: [INF] Applying service osd.None on host smithi102... 2021-11-20T08:54:58.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: audit 2021-11-20T08:54:58.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 2021-11-20T08:54: 2021-11-20T08:54:58.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 57.490574+0000 mon.smithi093 (mon. 2021-11-20T08:54:58.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 0) 306 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T08:54:58.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 2021-11-20T08:54:58.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: cluster 2021-11-20T08:54:58.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 2021-11-20T08: 2021-11-20T08:54:58.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 54:57.490634+0000 mon.smithi093 2021-11-20T08:54:58.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: (mon.0) 307 : cluster 2021-11-20T08:54:58.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T08:54:58.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: audit 2021 2021-11-20T08:54:58.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: -11-20T08:54:57 2021-11-20T08:54:58.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: .490937+0000 mon.smithi093 (mon. 2021-11-20T08:54:58.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 0) 308 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:58.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: audit 2021 2021-11-20T08:54:58.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: -11-20T08:54:57 2021-11-20T08:54:58.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: .491188+0000 mon.smithi093 (mon.0 2021-11-20T08:54:58.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: ) 309 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]: dispatch 2021-11-20T08:54:58.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 2021-11-20T08:54:58.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: cluster 2021-11-20T08:54:58.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: 2021-11-20T08 2021-11-20T08:54:58.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: :54:57.502519+0000 mgr.smithi093.qdikkm 2021-11-20T08:54:58.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: (mgr.14162) 109 : cluster 2021-11-20T08:54:58.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:58 smithi093 conmon[33522]: [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:54:59.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:59 smithi102 conmon[36472]: audit 2021-11-20T08:54:58.492528+0000 mon.smithi093 (mon.0) 310 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]': finished 2021-11-20T08:54:59.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:59 smithi102 conmon[36472]: cluster 2021-11-20T08:54:58.492582+0000 mon.smithi093 (mon.0) 311 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2021-11-20T08:54:59.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:59 smithi102 conmon[36472]: audit 2021-11-20T08:54:58.492706+0000 mon.smithi093 (mon.0) 312 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:59.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:59 smithi102 conmon[36472]: audit 2021-11-20T08:54:58.495727+0000 mon.smithi093 (mon.0) 313 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:59.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:54:59 smithi102 conmon[36472]: audit 2021-11-20T08:54:59.495532+0000 mon.smithi093 (mon.0) 314 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:59.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: audit 2021-11-20T08:54:58.492528+0000 mon.smithi093 ( 2021-11-20T08:54:59.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: mon.0) 310 : audit [INF] from='osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi093", "root=default"]}]': finished 2021-11-20T08:54:59.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: cluster 2021-11-20T08:54:58.492582+0000 mon.smithi093 (mon.0) 311 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2021-11-20T08:54:59.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: audit 2021-11-20T08:54:58.492706+0000 mon.smithi093 (mon.0) 312 2021-11-20T08:54:59.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:59.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:54:59.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: 11-20T08:54:58.495727+0000 2021-11-20T08:54:59.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: mon.smithi093 (mon.0) 313 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:54:59.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: 2021-11-20T08:54:59.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: audit 2021-11-20T08:54:59.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: 2021-11-20T08:54:59.495532+0000 mon.smithi093 ( 2021-11-20T08:54:59.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:54:59 smithi093 conmon[33522]: mon.0) 314 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:55:00.521 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: cluster 2021-11-20T08:54:57.961253+0000 osd.2 (osd.2) 2021-11-20T08:55:00.522 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:55:00.522 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: cluster 2021-11-20T08:54:57.961370+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:00.522 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: cluster 2021-11-20T08:54:59.502788+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:55:00.522 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: ) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:55:00.523 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:55:00.523 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 54:59.505760+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:00.523 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 315 : cluster [INF] osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] boot 2021-11-20T08:55:00.523 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T08:55:00.524 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: T08:54:59.505797+0000 mon.smithi093 (mon.0 2021-11-20T08:55:00.524 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: ) 316 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-11-20T08:55:00.524 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: audit 2021-11-20T08:54:59. 2021-11-20T08:55:00.524 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 506271+0000 mon.smithi093 (mon.0) 317 : audit 2021-11-20T08:55:00.524 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:55:00.525 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: audit 2021-11-20T08:54 2021-11-20T08:55:00.525 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: :59.734874+0000 mon.smithi102 (mon.1) 2 2021-11-20T08:55:00.525 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: : audit [INF] from='client.? 172.21.15.102:0/2226196541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37ba4275-7023-4f74-ba1f-7ad2f864476f"}]: dispatch 2021-11-20T08:55:00.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: cluster 2021-11-20T08:54:57.961253+0000 osd.2 (osd.2) 2021-11-20T08:55:00.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:55:00.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: cluster 2021-11-20T08:54:57.961370+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:00.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: cluster 2021-11-20T08:54:59.502788+0000 mgr.smithi093.qdikkm (mgr.14162) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T08:55:00.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: cluster 2021-11-20T08:54:59.505760+0000 mon.smithi093 (mon.0) 315 : cluster [INF] osd.2 [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] boot 2021-11-20T08:55:00.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: cluster 2021-11-20T08:54:59.505797+0000 mon.smithi093 (mon.0) 316 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-11-20T08:55:00.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: audit 2021-11-20T08:54:59.506271+0000 mon.smithi093 (mon.0) 317 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T08:55:00.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: audit 2021-11-20T08:54:59.734874+0000 mon.smithi102 (mon.1) 2 : audit [INF] from='client.? 172.21.15.102:0/2226196541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37ba4275-7023-4f74-ba1f-7ad2f864476f"}]: dispatch 2021-11-20T08:55:00.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: audit 2021-11-20T08:54:59.735272+0000 mon.smithi093 (mon.0) 318 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37ba4275-7023-4f74-ba1f-7ad2f864476f"}]: dispatch 2021-11-20T08:55:00.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: audit 2021-11-20T08:54:59.741328+0000 mon.smithi093 (mon.0) 319 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37ba4275-7023-4f74-ba1f-7ad2f864476f"}]': finished 2021-11-20T08:55:00.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: cluster 2021-11-20T08:54:59.741376+0000 mon.smithi093 (mon.0) 320 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T08:55:00.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: audit 2021-11-20T08:54:59.741474+0000 mon.smithi093 (mon.0) 321 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:00.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: audit 2021-11-20T08:54:59.892040+0000 mon.smithi093 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:00.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:00 smithi102 conmon[36472]: audit 2021-11-20T08:55:00.155058+0000 mon.smithi102 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.102:0/3731799213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:55:00.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: audit 2021-11-20T08:55:00.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 2021-11-20T08:54:59.735272 2021-11-20T08:55:00.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 318 : audit [INF] 2021-11-20T08:55:00.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37ba4275-7023-4f74-ba1f-7ad2f864476f"}]: dispatch 2021-11-20T08:55:00.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: audit 2021-11-20T08:54:59 2021-11-20T08:55:00.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: .741328+0000 mon.smithi093 (mon.0) 319 : audit 2021-11-20T08:55:00.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37ba4275-7023-4f74-ba1f-7ad2f864476f"}]': finished 2021-11-20T08:55:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:55:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 20T08:54:59.741376 2021-11-20T08:55:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 320 2021-11-20T08:55:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T08:55:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:55:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 11-20T08:54:59.741474+0000 2021-11-20T08:55:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: mon.smithi093 (mon.0) 321 : audit [DBG] 2021-11-20T08:55:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: audit 2021-11-20T08:55:00.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 2021-11-20T08:54:59.892040+0000 mon.smithi093 2021-11-20T08:55:00.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:00.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:55:00.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: 11-20T08:55:00.155058+0000 2021-11-20T08:55:00.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: mon.smithi102 (mon.1) 3 : audit [DBG] 2021-11-20T08:55:00.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:00 smithi093 conmon[33522]: from='client.? 172.21.15.102:0/3731799213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:55:02.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:01 smithi102 conmon[36472]: cluster 2021-11-20T08:55:00.895884+0000 mon.smithi093 (mon.0) 323 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T08:55:02.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:01 smithi102 conmon[36472]: audit 2021-11-20T08:55:00.895998+0000 mon.smithi093 (mon.0) 324 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:02.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:01 smithi102 conmon[36472]: cluster 2021-11-20T08:55:01.503031+0000 mgr.smithi093.qdikkm (mgr.14162) 111 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:02.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:01 smithi102 conmon[36472]: audit 2021-11-20T08:55:01.818924+0000 mon.smithi093 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:02.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:01 smithi093 conmon[33522]: cluster 2021-11-20T08:55:00.895884+0000 mon.smithi093 ( 2021-11-20T08:55:02.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:01 smithi093 conmon[33522]: mon.0) 323 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T08:55:02.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:01 smithi093 conmon[33522]: audit 2021-11-20T08:55:00.895998+0000 mon.smithi093 (mon.0) 324 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:02.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:01 smithi093 conmon[33522]: cluster 2021-11-20T08:55:01.503031+0000 mgr.smithi093.qdikkm (mgr.14162) 111 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:02.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:01 smithi093 conmon[33522]: audit 2021-11-20T08:55:01.818924+0000 mon.smithi093 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:03.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:03 smithi102 conmon[36472]: cluster 2021-11-20T08:55:03.503317+0000 mgr.smithi093.qdikkm (mgr.14162) 112 : cluster [DBG] 2021-11-20T08:55:03.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:03 smithi102 conmon[36472]: pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:03.836 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:03 smithi093 conmon[33522]: cluster 2021-11-20T08:55:03.503317+0000 mgr.smithi093.qdikkm (mgr.14162) 112 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:05.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:05 smithi102 conmon[36472]: audit 2021-11-20T08:55:04.608210+0000 mon.smithi093 (mon.0) 326 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:05.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:05 smithi102 conmon[36472]: audit 2021-11-20T08:55:04.610949+0000 mon.smithi093 (mon.0) 327 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:05.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:05 smithi102 conmon[36472]: cluster 2021-11-20T08:55:05.503664+0000 mgr.smithi093.qdikkm (mgr.14162) 113 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:05 smithi093 conmon[33522]: audit 2021-11-20T08:55:04.608210+0000 mon.smithi093 (mon.0) 326 : audit [INF] 2021-11-20T08:55:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:05 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:05 smithi093 conmon[33522]: audit 2021-11-20T08:55:04.610949+0000 mon.smithi093 (mon.0) 327 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:05 smithi093 conmon[33522]: cluster 2021-11-20T08:55:05.503664+0000 mgr.smithi093.qdikkm (mgr.14162) 113 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:06.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:06 smithi102 conmon[36472]: audit 2021-11-20T08:55:05.790701+0000 mon.smithi093 (mon. 2021-11-20T08:55:06.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:06 smithi102 conmon[36472]: 0) 328 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T08:55:06.876 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:06 smithi102 conmon[36472]: audit 2021-11-20T08:55:05.791543+0000 mon.smithi093 (mon.0) 329 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:06.877 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:06 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:05.792211+0000 mgr.smithi093.qdikkm (mgr.14162) 114 : cephadm [INF] Deploying daemon osd.3 on smithi102 2021-11-20T08:55:06.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:06 smithi093 conmon[33522]: audit 2021-11-20T08:55:05.790701+0000 mon.smithi093 (mon.0) 328 2021-11-20T08:55:06.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:06 smithi093 conmon[33522]: : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T08:55:06.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:06 smithi093 conmon[33522]: audit 2021-11-20T08:55:05.791543+0000 mon.smithi093 (mon.0) 329 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:06.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:06 smithi093 conmon[33522]: cephadm 2021-11-20T08:55:05.792211+0000 mgr.smithi093.qdikkm (mgr.14162) 114 : cephadm [INF] Deploying daemon osd.3 on smithi102 2021-11-20T08:55:07.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:07 smithi102 conmon[36472]: cluster 2021-11-20T08:55:07.504121+0000 2021-11-20T08:55:07.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:07 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:07.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:07 smithi093 conmon[33522]: cluster 2021-11-20T08:55:07.504121+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:55:07.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:07 smithi093 conmon[33522]: 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:08.867 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 3 on host 'smithi102' 2021-11-20T08:55:08.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.867+0000 7f7e5dffb700 1 -- 172.21.15.102:0/1550886973 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7e680008d0 con 0x7f7e5405c300 2021-11-20T08:55:08.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e5405c300 msgr2=0x7f7e5405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:08.869 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e5405c300 0x7f7e5405e7c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f7e680600a0 tx=0x7f7e6400b040).stop 2021-11-20T08:55:08.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 msgr2=0x7f7e6810bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:08.870 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e6810bf70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f7e58009f50 tx=0x7f7e58009b30).stop 2021-11-20T08:55:08.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 shutdown_connections 2021-11-20T08:55:08.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e5405c300 0x7f7e5405e7c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:08.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e680f5440 0x7f7e68109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:08.871 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 --2- 172.21.15.102:0/1550886973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e680f66e0 0x7f7e6810bf70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:08.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 >> 172.21.15.102:0/1550886973 conn(0x7f7e680f0ba0 msgr2=0x7f7e680fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:08.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 shutdown_connections 2021-11-20T08:55:08.872 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:08.869+0000 7f7e6e05b700 1 -- 172.21.15.102:0/1550886973 wait complete. 2021-11-20T08:55:09.209 DEBUG:teuthology.orchestra.run.smithi102:osd.3> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.3.service 2021-11-20T08:55:09.213 INFO:tasks.cephadm:Deploying osd.4 on smithi102 with /dev/vg_nvme/lv_3... 2021-11-20T08:55:09.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T08:55:09.242 INFO:journalctl@ceph.osd.3.smithi102.stdout:-- Logs begin at Sat 2021-11-20 08:41:07 UTC. -- 2021-11-20T08:55:09.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:09 smithi102 conmon[36472]: audit 2021-11-20T08:55:08.858547+0000 mon.smithi093 (mon.0) 330 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:09.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:09 smithi102 conmon[36472]: audit 2021-11-20T08:55: 2021-11-20T08:55:09.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:09 smithi102 conmon[36472]: 08.861995+0000 mon.smithi093 (mon.0) 331 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:09.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:09 smithi102 conmon[36472]: audit 2021-11-20T08:55:08.866951+0000 mon.smithi093 (mon.0) 332 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:09.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:09 smithi093 conmon[33522]: audit 2021-11-20T08:55:08.858547+0000 mon.smithi093 (mon.0 2021-11-20T08:55:09.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:09 smithi093 conmon[33522]: ) 330 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:09.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:09 smithi093 conmon[33522]: audit 2021-11-20T08:55:08.861995+0000 mon.smithi093 (mon.0) 331 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:09.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:09 smithi093 conmon[33522]: audit 2021-11-20T08:55:08.866951+0000 2021-11-20T08:55:09.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:09 smithi093 conmon[33522]: mon.smithi093 (mon.0) 332 : audit 2021-11-20T08:55:09.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:09 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:10.276 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:10 smithi102 conmon[36472]: audit 2021-11-20T08:55:08.868998+0000 mon.smithi093 (mon.0) 333 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T08:55:10.276 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:10 smithi102 conmon[36472]: cluster 2021-11-20T08:55:09.504453+0000 mgr.smithi093.qdikkm (mgr.14162) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:10.276 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:10 smithi102 conmon[36472]: audit 2021-11-20T08:55:09.868785+0000 mon.smithi093 (mon.0) 334 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T08:55:10.277 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:10 smithi102 conmon[36472]: cluster 2021-11-20T08:55:09.868844+0000 mon.smithi093 (mon.0) 335 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-20T08:55:10.277 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:10 smithi102 conmon[36472]: audit 2021-11-20T08:55:09.868994+0000 mon.smithi093 (mon.0) 336 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:10.277 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:10 smithi102 conmon[36472]: audit 2021-11-20T08:55:09.869712+0000 mon.smithi093 (mon.0) 337 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T08:55:10.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: audit 2021-11-20T08:55:08.868998+0000 mon.smithi093 2021-11-20T08:55:10.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: (mon.0) 333 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T08:55:10.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: cluster 2021-11-20T08:55:09.504453+0000 mgr.smithi093.qdikkm (mgr.14162) 116 : cluster 2021-11-20T08:55:10.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: [DBG] pgmap v79: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:10.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: audit 2021-11-20T08 2021-11-20T08:55:10.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: :55:09.868785+0000 mon.smithi093 (mon.0) 334 : audit 2021-11-20T08:55:10.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T08:55:10.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:55:10.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: 55:09.868844+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:10.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: 335 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-20T08:55:10.466 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:55:10.466 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: 20T08:55:09.868994+0000 mon.smithi093 2021-11-20T08:55:10.466 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: (mon.0) 336 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:10.467 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: 2021-11-20T08:55:10.467 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: audit 2021 2021-11-20T08:55:10.468 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: -11-20T08:55:09.869712+0000 mon.smithi093 ( 2021-11-20T08:55:10.468 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:10 smithi093 conmon[33522]: mon.0) 337 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T08:55:11.250 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T08:55:11.251 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T08:55:11.251 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2021-11-20T08:55:11.251 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2021-11-20T08:55:11.251 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145532 s, 721 MB/s 2021-11-20T08:55:11.252 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2021-11-20T08:55:11.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2021-11-20T08:55:12.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:11 smithi102 conmon[36472]: audit 2021-11-20T08:55:10.870039+0000 mon.smithi093 (mon.0) 338 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T08:55:12.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:11 smithi102 conmon[36472]: cluster 2021-11-20T08:55:10.870118+0000 mon.smithi093 (mon.0) 339 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-20T08:55:12.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:11 smithi102 conmon[36472]: audit 2021-11-20T08:55:10.870246+0000 mon.smithi093 (mon.0) 340 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:12.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:11 smithi102 conmon[36472]: cluster 2021-11-20T08:55:11.504736+0000 mgr.smithi093.qdikkm (mgr.14162) 117 : cluster [DBG] pgmap v82: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:12.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:11 smithi102 conmon[36472]: audit 2021-11-20T08:55:11.789023 2021-11-20T08:55:12.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:11 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 341 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:12.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: audit 2021-11-20T08:55:10.870039+0000 mon.smithi093 (mon.0) 338 : 2021-11-20T08:55:12.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T08:55:12.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: cluster 2021-11-20T08:55:10.870118+0000 mon.smithi093 (mon.0) 339 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-20T08:55:12.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: audit 2021-11-20T08:55:10. 2021-11-20T08:55:12.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: 870246+0000 mon.smithi093 (mon.0) 340 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:12.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: cluster 2021-11-20T08:55:11.504736+0000 2021-11-20T08:55:12.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 117 : cluster [DBG] pgmap v82: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:12.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: audit 2021 2021-11-20T08:55:12.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:11 smithi093 conmon[33522]: -11-20T08:55:11.789023+0000 mon.smithi093 (mon.0) 341 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:13.099 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[43029]: debug 2021-11-20T08:55:12.723+0000 7f51c00c8080 -1 osd.3 0 log_to_monitors {default=true} 2021-11-20T08:55:13.100 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[36472]: cluster 2021-11-20T08:55:11.876226+0000 mon.smithi093 (mon.0 2021-11-20T08:55:13.100 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[36472]: ) 342 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-20T08:55:13.100 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[36472]: audit 2021-11-20T08:55:11.876340+0000 mon.smithi093 (mon.0) 343 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:13.100 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[36472]: audit 2021-11-20T08:55:12.726181+0000 mon.smithi102 (mon.1) 2021-11-20T08:55:13.101 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[36472]: 4 : audit [INF] from='osd.3 [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T08:55:13.101 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[36472]: audit 2021-11-20T08:55:12.726588 2021-11-20T08:55:13.101 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:12 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 344 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T08:55:13.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:12 smithi093 conmon[33522]: cluster 2021-11-20T08:55:11.876226+0000 mon.smithi093 (mon.0) 342 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-20T08:55:13.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:12 smithi093 conmon[33522]: audit 2021-11-20T08:55:11.876340+0000 mon.smithi093 (mon.0) 343 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:13.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:12 smithi093 conmon[33522]: audit 2021-11-20T08:55:12.726181+0000 mon.smithi102 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T08:55:13.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:12 smithi093 conmon[33522]: audit 2021-11-20T08:55:12.726588+0000 mon.smithi093 (mon.0) 344 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T08:55:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 -- 172.21.15.102:0/3803136533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f6700 msgr2=0x7f37dc0f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 --2- 172.21.15.102:0/3803136533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f6700 0x7f37dc0f6b60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f37d0004660 tx=0x7f37d0009b30).stop 2021-11-20T08:55:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 -- 172.21.15.102:0/3803136533 shutdown_connections 2021-11-20T08:55:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 --2- 172.21.15.102:0/3803136533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f6700 0x7f37dc0f6b60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 --2- 172.21.15.102:0/3803136533 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37dc0f5460 0x7f37dc0f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 -- 172.21.15.102:0/3803136533 >> 172.21.15.102:0/3803136533 conn(0x7f37dc0f0bc0 msgr2=0x7f37dc0f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 -- 172.21.15.102:0/3803136533 shutdown_connections 2021-11-20T08:55:13.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.636+0000 7f37e146e700 1 -- 172.21.15.102:0/3803136533 wait complete. 2021-11-20T08:55:13.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37e146e700 1 Processor -- start 2021-11-20T08:55:13.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37e146e700 1 -- start start 2021-11-20T08:55:13.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37e146e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f5460 0x7f37dc105d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:13.639 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37e146e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37dc0f6700 0x7f37dc106220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:13.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37e146e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc1067d0 con 0x7f37dc0f6700 2021-11-20T08:55:13.640 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37e146e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc106fd0 con 0x7f37dc0f5460 2021-11-20T08:55:13.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37daffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f5460 0x7f37dc105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:13.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37daffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f5460 0x7f37dc105d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53742/0 (socket says 172.21.15.102:53742) 2021-11-20T08:55:13.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37daffd700 1 -- 172.21.15.102:0/645434222 learned_addr learned my addr 172.21.15.102:0/645434222 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:55:13.641 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37da7fc700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37dc0f6700 0x7f37dc106220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:13.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37daffd700 1 -- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37dc0f6700 msgr2=0x7f37dc106220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:13.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37daffd700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37dc0f6700 0x7f37dc106220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:13.642 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.637+0000 7f37daffd700 1 -- 172.21.15.102:0/645434222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d0005040 con 0x7f37dc0f5460 2021-11-20T08:55:13.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.638+0000 7f37daffd700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f5460 0x7f37dc105d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f37cc002700 tx=0x7f37cc009d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:13.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.638+0000 7f37c37fe700 1 -- 172.21.15.102:0/645434222 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37cc0079c0 con 0x7f37dc0f5460 2021-11-20T08:55:13.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.638+0000 7f37c37fe700 1 -- 172.21.15.102:0/645434222 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37cc00f040 con 0x7f37dc0f5460 2021-11-20T08:55:13.643 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.638+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37dc1072b0 con 0x7f37dc0f5460 2021-11-20T08:55:13.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.638+0000 7f37c37fe700 1 -- 172.21.15.102:0/645434222 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37cc014860 con 0x7f37dc0f5460 2021-11-20T08:55:13.644 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.638+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37dc107870 con 0x7f37dc0f5460 2021-11-20T08:55:13.645 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.639+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37dc059100 con 0x7f37dc0f5460 2021-11-20T08:55:13.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.639+0000 7f37c37fe700 1 -- 172.21.15.102:0/645434222 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f37cc01b060 con 0x7f37dc0f5460 2021-11-20T08:55:13.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.639+0000 7f37c37fe700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f37c405c300 0x7f37c405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:13.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.639+0000 7f37c37fe700 1 -- 172.21.15.102:0/645434222 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3282+0+0 (secure 0 0 0) 0x7f37cc0754e0 con 0x7f37dc0f5460 2021-11-20T08:55:13.646 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.640+0000 7f37da7fc700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f37c405c300 0x7f37c405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:13.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.640+0000 7f37da7fc700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f37c405c300 0x7f37c405e7c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f37d0009f50 tx=0x7f37d0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:13.647 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.643+0000 7f37c37fe700 1 -- 172.21.15.102:0/645434222 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37cc0489d0 con 0x7f37dc0f5460 2021-11-20T08:55:13.831 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:13.832+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f37dc0ff6c0 con 0x7f37c405c300 2021-11-20T08:55:14.533 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[43029]: debug 2021-11-20T08:55:14.499+0000 7f51b6639700 -1 osd.3 0 waiting for initial osdmap 2021-11-20T08:55:14.534 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[43029]: debug 2021-11-20T08:55:14.505+0000 7f51b15b2700 -1 osd.3 26 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T08:55:14.534 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.279534+0000 mon.smithi093 (mon.0) 345 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:14.534 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.494114+0000 mon.smithi093 (mon.0) 346 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T08:55:14.535 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: cluster 2021-11-20T08:55:13.494177+0000 mon.smithi093 (mon.0) 347 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-20T08:55:14.536 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.494300+0000 mon.smithi093 (mon.0) 348 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:14.536 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.494714+0000 mon.smithi102 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:14.536 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.494952+0000 mon.smithi093 (mon.0) 349 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:14.536 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: cluster 2021-11-20T08:55:13.505097+0000 mgr.smithi093.qdikkm (mgr.14162) 118 : cluster 2021-11-20T08:55:14.537 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: [DBG] pgmap v85: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:14.537 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.834406+0000 mon.smithi093 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:55:14.537 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.835693+0000 mon.smithi093 (mon.0) 351 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:55:14.537 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:14 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.836273+0000 mon.smithi093 (mon.0) 352 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:14.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.279534+0000 mon.smithi093 (mon.0) 345 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:14.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: 2021-11-20T08:55:14.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.494114+0000 mon.smithi093 (mon.0) 346 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T08:55:14.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: cluster 2021-11-20T08:55:13.494177+0000 mon.smithi093 (mon.0) 347 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-20T08:55:14.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.494300+0000 mon.smithi093 (mon.0) 348 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:14.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.494714+0000 mon.smithi102 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:14.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.494952+0000 mon.smithi093 (mon.0) 349 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:14.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: cluster 2021-11-20T08:55:13.505097+0000 mgr.smithi093.qdikkm (mgr.14162) 118 : cluster [DBG] pgmap v85: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:14.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.834406+0000 mon.smithi093 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:55:14.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.835693+0000 mon.smithi093 (mon.0) 351 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:55:14.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:14 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.836273+0000 mon.smithi093 (mon.0) 352 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:15.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:15 smithi102 conmon[36472]: audit 2021-11-20T08:55:13.833669+0000 mgr.smithi093.qdikkm (mgr.14162) 119 : audit [DBG] from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:55:15.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:15 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:13.835481+0000 mgr.smithi093.qdikkm (mgr.14162) 120 : cephadm [INF] Applying service osd.None on host smithi102... 2021-11-20T08:55:15.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:15 smithi102 conmon[36472]: audit 2021-11-20T08:55:14.496039+0000 mon.smithi093 (mon.0) 353 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2021-11-20T08:55:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:15 smithi102 conmon[36472]: cluster 2021-11-20T08:55:14.496179+0000 mon.smithi093 (mon.0) 354 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2021-11-20T08:55:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:15 smithi102 conmon[36472]: audit 2021-11-20T08:55:14.496561+0000 mon.smithi093 (mon.0) 355 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:15 smithi102 conmon[36472]: audit 2021-11-20T08:55:14.499960+0000 mon.smithi093 (mon.0) 356 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:15.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: audit 2021-11-20T08:55:13.833669+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:55:15.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: mgr.14162) 119 : audit [DBG] from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:55:15.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: cephadm 2021-11-20T08:55:13.835481+0000 mgr.smithi093.qdikkm (mgr.14162) 120 : cephadm [INF] Applying service osd.None on host smithi102... 2021-11-20T08:55:15.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: audit 2021-11-20T08:55:14.496039+0000 mon.smithi093 (mon.0) 353 : audit [INF] 2021-11-20T08:55:15.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2021-11-20T08:55:15.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: cluster 2021-11-20T08:55:14.496179+0000 mon.smithi093 2021-11-20T08:55:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: (mon.0) 354 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2021-11-20T08:55:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:55:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: -20T08:55:14.496561+0000 mon.smithi093 (mon.0) 355 : audit 2021-11-20T08:55:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: audit 2021-11-20T08:55:14 2021-11-20T08:55:15.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: .499960+0000 mon.smithi093 (mon.0) 356 : audit [DBG] 2021-11-20T08:55:15.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:15 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:16.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: cluster 2021-11-20T08:55:13. 2021-11-20T08:55:16.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: 733509+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:55:16.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: cluster 2021-11-20T08:55:13.733622+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:16.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:15.499503+0000 mon.smithi093 (mon.0) 357 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: cluster 2021-11-20T08:55:15.505361+0000 mgr.smithi093.qdikkm (mgr.14162) 121 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: cluster 2021-11-20T08:55:15.507117+0000 mon.smithi093 (mon.0) 358 : cluster [INF] osd.3 [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] boot 2021-11-20T08:55:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: cluster 2021-11-20T08:55:15.507180+0000 mon.smithi093 (mon.0) 359 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T08:55:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:15.507337+0000 mon.smithi093 (mon.0) 360 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:16.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:15.702101+0000 mon.smithi093 (mon.0) 361 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:16.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:15.705752+0000 mon.smithi093 (mon.0) 362 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:16.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:15.708957+0000 mon.smithi093 (mon.0) 363 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:16.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:16.220985+0000 mon.smithi102 (mon.1) 6 : audit [INF] from='client.? 172.21.15.102:0/3242693762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9"}]: dispatch 2021-11-20T08:55:16.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:16.221560+0000 mon.smithi093 (mon.0) 364 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9"}]: dispatch 2021-11-20T08:55:16.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021-11-20T08:55:16.227763+0000 mon.smithi093 (mon.0) 365 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9"}]': finished 2021-11-20T08:55:16.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T08:55:16.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: 20T08:55:16. 2021-11-20T08:55:16.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: 227859+0000 mon.smithi093 (mon.0 2021-11-20T08:55:16.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: ) 366 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-20T08:55:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:55:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: 11-20T08:55:16.227997 2021-11-20T08:55:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 2021-11-20T08:55:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:16 smithi102 conmon[36472]: 367 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:16.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: cluster 2021-11-20T08:55:13.733509+0000 2021-11-20T08:55:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:55:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: cluster 2021-11-20T08:55:13.733622+0000 osd.3 (osd.3 2021-11-20T08:55:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021-11-20T08:55:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 2021-11-20T08:55:15.499503+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 357 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:55:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 11-20T08:55:15.505361+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:55:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: mgr.14162) 121 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T08:55:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: cluster 2021-11-20T08:55:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 2021-11-20T08:55:15.507117 2021-11-20T08:55:16.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 358 : cluster [INF] 2021-11-20T08:55:16.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: osd.3 [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] boot 2021-11-20T08:55:16.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: cluster 2021-11-20T08 2021-11-20T08:55:16.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: :55:15.507180+0000 mon.smithi093 (mon.0) 359 2021-11-20T08:55:16.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T08:55:16.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:55:16.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 11-20T08:55:15.507337+0000 mon.smithi093 2021-11-20T08:55:16.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: (mon.0) 360 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T08:55:16.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:55:16.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 11-20T08:55:15.702101+0000 mon.smithi093 (mon 2021-11-20T08:55:16.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: .0) 361 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:16.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:55:16.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 11-20T08:55:15.705752+0000 mon.smithi093 2021-11-20T08:55:16.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: (mon.0) 362 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:16.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021-11-20T08:55:16.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 2021-11-20T08 2021-11-20T08:55:16.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: :55:15.708957+0000 mon.smithi093 (mon. 2021-11-20T08:55:16.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 0) 363 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:16.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021-11-20T08:55:16.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 2021-11-20T08:55:16.220985+0000 2021-11-20T08:55:16.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: mon.smithi102 (mon.1) 6 : audit [INF] from='client.? 172.21.15.102:0/3242693762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9"}]: dispatch 2021-11-20T08:55:16.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 2021-11-20T08:55:16.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021-11-20T08:55: 2021-11-20T08:55:16.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 16.221560+0000 mon.smithi093 (mon.0) 364 : 2021-11-20T08:55:16.972 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9"}]: dispatch 2021-11-20T08:55:16.972 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:55:16.972 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 08:55:16.227763+0000 mon.smithi093 (mon. 2021-11-20T08:55:16.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 0) 365 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9"}]': finished 2021-11-20T08:55:16.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: cluster 2021-11-20T08:55:16.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 2021-11-20T08:55 2021-11-20T08:55:16.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: :16.227859+0000 mon.smithi093 (mon.0) 366 2021-11-20T08:55:16.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-20T08:55:16.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: audit 2021-11-20T08:55:16.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 2021-11-20T08:55:16. 2021-11-20T08:55:16.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: 227997+0000 mon.smithi093 (mon.0) 367 : audit 2021-11-20T08:55:16.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:16 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:17.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:15.695424+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:55:17.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: 122 : cephadm [INF] Detected new or changed devices on smithi102 2021-11-20T08:55:17.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: audit 2021-11-20T08:55:16.633625+0000 mon.smithi102 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.102:0/4075806616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:55:17.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: cluster 2021-11-20T08:55:17.226649+0000 mon.smithi093 2021-11-20T08:55:17.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: (mon.0) 368 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T08:55:17.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T08:55:17.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: 08:55:17.226773+0000 mon.smithi093 (mon.0) 369 : audit [DBG] 2021-11-20T08:55:17.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:17 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:17.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:17 smithi093 conmon[33522]: cephadm 2021-11-20T08:55:15.695424+0000 mgr.smithi093.qdikkm (mgr.14162) 122 : cephadm [INF] Detected new or changed devices on smithi102 2021-11-20T08:55:17.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:17 smithi093 conmon[33522]: audit 2021-11-20T08:55:16.633625+0000 mon.smithi102 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.102:0/4075806616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:55:17.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:17 smithi093 conmon[33522]: cluster 2021-11-20T08:55:17.226649+0000 mon.smithi093 (mon.0) 368 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T08:55:17.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:17 smithi093 conmon[33522]: audit 2021-11-20T08:55:17.226773+0000 mon.smithi093 (mon.0) 369 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:18.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:18 smithi102 conmon[36472]: cluster 2021-11-20T08:55:17.505612+0000 mgr.smithi093.qdikkm (mgr.14162) 123 : cluster [DBG] 2021-11-20T08:55:18.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:18 smithi102 conmon[36472]: pgmap v91: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:18.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:18 smithi093 conmon[33522]: cluster 2021-11-20T08:55:17.505612+0000 2021-11-20T08:55:18.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:18 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 123 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:20.852 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:20 smithi102 conmon[36472]: cluster 2021-11-20T08:55:19.506087+0000 2021-11-20T08:55:20.853 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:20 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 124 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:20.853 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:20 smithi102 conmon[36472]: audit 2021-11-20T08:55:20.310498+0000 mon.smithi093 (mon.0) 370 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:20.853 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:20 smithi102 conmon[36472]: audit 2021-11-20T08:55:20.313426+0000 mon.smithi093 (mon.0) 371 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:20 smithi093 conmon[33522]: cluster 2021-11-20T08:55:19.506087+0000 2021-11-20T08:55:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:20 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 124 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:20 smithi093 conmon[33522]: audit 2021-11-20T08:55:20.310498+0000 mon.smithi093 (mon.0) 370 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:20.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:20 smithi093 conmon[33522]: audit 2021-11-20T08:55:20.313426+0000 mon.smithi093 (mon.0) 371 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:22.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:22 smithi102 conmon[36472]: cluster 2021-11-20T08:55:21.506453+0000 mgr.smithi093.qdikkm (mgr.14162) 125 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:22.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:22 smithi102 conmon[36472]: audit 2021-11-20T08:55:22.381621+0000 mon.smithi093 (mon.0) 372 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T08:55:22.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:22 smithi102 conmon[36472]: audit 2021-11-20T08:55:22.382542+0000 mon.smithi093 (mon.0) 373 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:22 smithi093 conmon[33522]: cluster 2021-11-20T08:55:21.506453+0000 mgr.smithi093.qdikkm (mgr.14162) 125 2021-11-20T08:55:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:22 smithi093 conmon[33522]: : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:22 smithi093 conmon[33522]: audit 2021-11-20T08:55:22.381621+0000 mon.smithi093 (mon.0) 372 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T08:55:22.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:22 smithi093 conmon[33522]: audit 2021-11-20T08:55:22.382542+0000 mon.smithi093 (mon.0) 373 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:23.800 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:23 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:22.383279+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:55:23.800 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:23 smithi102 conmon[36472]: 126 : cephadm [INF] Deploying daemon osd.4 on smithi102 2021-11-20T08:55:23.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:23 smithi093 conmon[33522]: cephadm 2021-11-20T08:55:22.383279+0000 mgr.smithi093.qdikkm 2021-11-20T08:55:23.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:23 smithi093 conmon[33522]: (mgr.14162) 126 : cephadm [INF] Deploying daemon osd.4 on smithi102 2021-11-20T08:55:24.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:24 smithi102 conmon[36472]: cluster 2021-11-20T08:55:23.506834+0000 mgr.smithi093.qdikkm 2021-11-20T08:55:24.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:24 smithi102 conmon[36472]: (mgr.14162) 127 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:24 smithi093 conmon[33522]: cluster 2021-11-20T08:55:23.506834+0000 mgr.smithi093.qdikkm (mgr.14162) 127 2021-11-20T08:55:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:24 smithi093 conmon[33522]: : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:25.278 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.278+0000 7f37c37fe700 1 -- 172.21.15.102:0/645434222 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f37dc0ff6c0 con 0x7f37c405c300 2021-11-20T08:55:25.279 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 4 on host 'smithi102' 2021-11-20T08:55:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f37c405c300 msgr2=0x7f37c405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f37c405c300 0x7f37c405e7c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f37d0009f50 tx=0x7f37d0009b30).stop 2021-11-20T08:55:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f5460 msgr2=0x7f37dc105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f5460 0x7f37dc105d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f37cc002700 tx=0x7f37cc009d80).stop 2021-11-20T08:55:25.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 shutdown_connections 2021-11-20T08:55:25.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f37c405c300 0x7f37c405e7c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:25.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37dc0f5460 0x7f37dc105d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:25.281 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 --2- 172.21.15.102:0/645434222 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37dc0f6700 0x7f37dc106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:25.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 >> 172.21.15.102:0/645434222 conn(0x7f37dc0f0bc0 msgr2=0x7f37dc0fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:25.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 shutdown_connections 2021-11-20T08:55:25.282 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:25.281+0000 7f37e146e700 1 -- 172.21.15.102:0/645434222 wait complete. 2021-11-20T08:55:25.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:25 smithi102 conmon[36472]: audit 2021-11-20T08:55:25.267754+0000 mon.smithi093 (mon.0) 374 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:25.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:25 smithi102 conmon[36472]: audit 2021-11-20T08:55:25.272101+0000 mon.smithi093 (mon.0) 375 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:25.646 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:25 smithi102 conmon[36472]: audit 2021-11-20T08:55:25.277391+0000 mon.smithi093 (mon.0) 376 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:25.707 DEBUG:teuthology.orchestra.run.smithi102:osd.4> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.4.service 2021-11-20T08:55:25.711 INFO:tasks.cephadm:Deploying osd.5 on smithi102 with /dev/vg_nvme/lv_2... 2021-11-20T08:55:25.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T08:55:25.741 INFO:journalctl@ceph.osd.4.smithi102.stdout:-- Logs begin at Sat 2021-11-20 08:41:07 UTC. -- 2021-11-20T08:55:25.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:25 smithi093 conmon[33522]: audit 2021-11-20T08:55:25.267754+0000 2021-11-20T08:55:25.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:25 smithi093 conmon[33522]: mon.smithi093 (mon.0) 374 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:25.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:25 smithi093 conmon[33522]: audit 2021-11-20T08:55:25.272101+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:25.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:25 smithi093 conmon[33522]: 375 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:25 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:55:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:25 smithi093 conmon[33522]: 11-20T08:55:25.277391+0000 mon.smithi093 (mon.0) 376 : 2021-11-20T08:55:25.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:25 smithi093 conmon[33522]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:26.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:26 smithi102 conmon[36472]: cluster 2021-11-20T08:55:25.507081+0000 mgr.smithi093.qdikkm (mgr.14162) 128 : 2021-11-20T08:55:26.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:26 smithi102 conmon[36472]: cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:26.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:26 smithi093 conmon[33522]: cluster 2021-11-20T08:55:25.507081+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:55:26.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:26 smithi093 conmon[33522]: 128 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:27.684 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T08:55:27.684 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T08:55:27.684 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2021-11-20T08:55:27.685 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2021-11-20T08:55:27.685 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130334 s, 805 MB/s 2021-11-20T08:55:27.685 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2021-11-20T08:55:27.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:27 smithi102 conmon[36472]: cluster 2021-11-20T08:55:27.507351+0000 mgr.smithi093.qdikkm (mgr.14162) 129 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:27.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:27 smithi093 conmon[33522]: cluster 2021-11-20T08:55:27.507351+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:55:27.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:27 smithi093 conmon[33522]: .14162) 129 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:28.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2021-11-20T08:55:29.306 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 08:55:29 smithi102 conmon[47885]: debug 2021-11-20T08:55:29.254+0000 7f669e7be080 -1 osd.4 0 log_to_monitors {default=true} 2021-11-20T08:55:29.601 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:29 smithi102 conmon[36472]: audit 2021-11-20T08:55:28.530055+0000 mon.smithi093 (mon.0) 377 2021-11-20T08:55:29.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:29 smithi102 conmon[36472]: : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:29.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:29 smithi102 conmon[36472]: audit 2021-11-20T08:55:29.257475+0000 mon.smithi102 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T08:55:29.602 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:29 smithi102 conmon[36472]: audit 2021-11-20T08:55:29.257851+0000 mon.smithi093 (mon.0) 378 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T08:55:29.603 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:29 smithi102 conmon[36472]: cluster 2021-11-20T08:55:29.507720+0000 mgr.smithi093.qdikkm (mgr.14162) 130 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:29.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: audit 2021-11-20T08 2021-11-20T08:55:29.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: :55:28.530055+0000 mon.smithi093 (mon.0) 377 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:29.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: audit 2021-11-20T08:55:29.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: 2021-11-20T08:55:29. 2021-11-20T08:55:29.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: 257475+0000 mon.smithi102 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T08:55:29.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:55:29.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: 55:29.257851+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:29.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: 378 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T08:55:29.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:55:29.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: 11-20T08:55:29.507720+0000 2021-11-20T08:55:29.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:29 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 130 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:30.082 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 -- 172.21.15.102:0/154534252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f51a0 msgr2=0x7fcb4c0f7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:30.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 --2- 172.21.15.102:0/154534252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f51a0 0x7fcb4c0f7920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fcb40004660 tx=0x7fcb40009b30).stop 2021-11-20T08:55:30.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 -- 172.21.15.102:0/154534252 shutdown_connections 2021-11-20T08:55:30.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 --2- 172.21.15.102:0/154534252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f51a0 0x7fcb4c0f7920 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0x7fcb40004660 tx=0x7fcb40009b30).stop 2021-11-20T08:55:30.083 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 --2- 172.21.15.102:0/154534252 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb4c0f4800 0x7fcb4c0f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:30.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 -- 172.21.15.102:0/154534252 >> 172.21.15.102:0/154534252 conn(0x7fcb4c0f0320 msgr2=0x7fcb4c0f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:30.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 -- 172.21.15.102:0/154534252 shutdown_connections 2021-11-20T08:55:30.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.081+0000 7fcb5090f700 1 -- 172.21.15.102:0/154534252 wait complete. 2021-11-20T08:55:30.084 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb5090f700 1 Processor -- start 2021-11-20T08:55:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb5090f700 1 -- start start 2021-11-20T08:55:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb5090f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f4800 0x7fcb4c0f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb5090f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb4c0f5830 0x7fcb4c0f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb5090f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0f61a0 con 0x7fcb4c0f5830 2021-11-20T08:55:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb5090f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0f62e0 con 0x7fcb4c0f4800 2021-11-20T08:55:30.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb4ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f4800 0x7fcb4c0f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:30.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.082+0000 7fcb4ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f4800 0x7fcb4c0f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53804/0 (socket says 172.21.15.102:53804) 2021-11-20T08:55:30.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb4ad9d700 1 -- 172.21.15.102:0/3257782380 learned_addr learned my addr 172.21.15.102:0/3257782380 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:55:30.086 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb4a59c700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb4c0f5830 0x7fcb4c0f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:30.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb4ad9d700 1 -- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb4c0f5830 msgr2=0x7fcb4c0f5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:30.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb4ad9d700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb4c0f5830 0x7fcb4c0f5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:30.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb4ad9d700 1 -- 172.21.15.102:0/3257782380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb40005040 con 0x7fcb4c0f4800 2021-11-20T08:55:30.087 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb4a59c700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb4c0f5830 0x7fcb4c0f5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:55:30.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb4ad9d700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f4800 0x7fcb4c0f73b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fcb3c002700 tx=0x7fcb3c009d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:30.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb337fe700 1 -- 172.21.15.102:0/3257782380 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c009320 con 0x7fcb4c0f4800 2021-11-20T08:55:30.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c0f6590 con 0x7fcb4c0f4800 2021-11-20T08:55:30.088 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb337fe700 1 -- 172.21.15.102:0/3257782380 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb3c00f040 con 0x7fcb4c0f4800 2021-11-20T08:55:30.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb337fe700 1 -- 172.21.15.102:0/3257782380 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c014900 con 0x7fcb4c0f4800 2021-11-20T08:55:30.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.083+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c10f270 con 0x7fcb4c0f4800 2021-11-20T08:55:30.089 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.084+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c045ad0 con 0x7fcb4c0f4800 2021-11-20T08:55:30.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.084+0000 7fcb337fe700 1 -- 172.21.15.102:0/3257782380 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcb3c0077b0 con 0x7fcb4c0f4800 2021-11-20T08:55:30.090 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.085+0000 7fcb337fe700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcb3405c300 0x7fcb3405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:30.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.085+0000 7fcb337fe700 1 -- 172.21.15.102:0/3257782380 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3824+0+0 (secure 0 0 0) 0x7fcb3c075b90 con 0x7fcb4c0f4800 2021-11-20T08:55:30.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.085+0000 7fcb4a59c700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcb3405c300 0x7fcb3405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:30.091 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.085+0000 7fcb4a59c700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcb3405c300 0x7fcb3405e7c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcb40000c00 tx=0x7fcb40009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:30.092 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.088+0000 7fcb337fe700 1 -- 172.21.15.102:0/3257782380 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb3c048e70 con 0x7fcb4c0f4800 2021-11-20T08:55:30.276 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:30.278+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb4c057f10 con 0x7fcb3405c300 2021-11-20T08:55:30.752 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[47885]: debug 2021-11-20T08:55:30.542+0000 7f6694d2f700 -1 osd.4 0 waiting for initial osdmap 2021-11-20T08:55:30.753 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[47885]: debug 2021-11-20T08:55:30.551+0000 7f668d4a3700 -1 osd.4 31 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T08:55:30.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:29.537211+0000 mon.smithi093 (mon.0) 379 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T08:55:30.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: cluster 2021-11-20T08:55:29.537269+0000 mon.smithi093 (mon.0) 380 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T08:55:30.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:29.537517+0000 mon.smithi093 (mon.0) 381 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:30.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:29.537786+0000 mon.smithi102 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:30.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:29.538019+0000 mon.smithi093 (mon.0) 382 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:30.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:29.728103+0000 mon.smithi093 (mon.0) 383 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:30.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:30.279268+0000 mgr.smithi093.qdikkm (mgr.14162) 131 : audit [DBG] from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:55:30.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:30.280104+0000 mon.smithi093 (mon.0) 384 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:55:30.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:30.281226+0000 mgr.smithi093.qdikkm (mgr.14162) 132 : cephadm [INF] Applying service osd.None on host smithi102... 2021-11-20T08:55:30.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:30.281448+0000 mon.smithi093 (mon.0) 385 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:55:30.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:30 smithi102 conmon[36472]: audit 2021-11-20T08:55:30.282002+0000 mon.smithi093 (mon.0) 386 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:30.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T08:55:29.537211+0000 mon.smithi093 (mon.0) 379 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T08:55:30.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: cluster 2021-11-20T08:55 2021-11-20T08:55:30.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: :29.537269+0000 mon.smithi093 (mon.0) 380 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T08:55:30.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T08:55:29.537517+0000 mon.smithi093 (mon.0) 381 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:30.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T08:55:29.537786+0000 mon.smithi102 (mon. 2021-11-20T08:55:30.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 1) 9 : audit [INF] from='osd.4 [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:30.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:55:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 08:55:29.538019+0000 mon.smithi093 (mon.0) 382 : audit [INF] 2021-11-20T08:55:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T08:55:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 2021-11-20T08:55:29.728103+0000 mon.smithi093 (mon.0) 383 : audit 2021-11-20T08:55:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:55:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 55:30.279268+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:55:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 131 : audit [DBG] from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:55:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T08 2021-11-20T08:55:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: :55:30.280104+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 384 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T08:55:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: cephadm 2021-11- 2021-11-20T08:55:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 20T08:55:30.281226+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:55:30.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: mgr.14162) 132 : cephadm [INF] Applying service osd.None on host smithi102... 2021-11-20T08:55:30.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:55:30.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: 11-20T08:55:30.281448+0000 2021-11-20T08:55:30.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: mon.smithi093 (mon.0) 385 : audit [INF] 2021-11-20T08:55:30.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T08:55:30.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: audit 2021-11-20T08:55:30 2021-11-20T08:55:30.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: .282002+0000 mon.smithi093 (mon.0) 386 : audit 2021-11-20T08:55:30.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:30 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:31.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: audit 2021-11-20T08:55:30.539138+0000 mon.smithi093 (mon.0) 387 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2021-11-20T08:55:31.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: cluster 2021-11-20T08:55:30.539206+0000 mon.smithi093 (mon.0) 388 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2021-11-20T08:55:31.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: audit 2021-11-20T08:55:30.539513+0000 mon.smithi093 (mon.0) 389 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:31.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: audit 2021-11-20T08:55:30.542816+0000 mon.smithi093 (mon.0) 390 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:31.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: cluster 2021-11-20T08:55:31.508066+0000 mgr.smithi093.qdikkm (mgr.14162) 133 : cluster [DBG] pgmap v100: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:31.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: cluster 2021-11-20T08:55:31.537499+0000 mon.smithi093 (mon.0) 391 : cluster [INF] osd.4 [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] boot 2021-11-20T08:55:31.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: cluster 2021-11-20T08:55:31.537544+0000 mon.smithi093 (mon.0) 392 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-11-20T08:55:31.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:31 smithi102 conmon[36472]: audit 2021-11-20T08:55:31.537703+0000 mon.smithi093 (mon.0) 393 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:31.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: audit 2021-11-20T08:55:30.539138+0000 mon.smithi093 (mon 2021-11-20T08:55:31.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: .0) 387 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2021-11-20T08:55:31.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: cluster 2021-11-20T08:55:30.539206+0000 mon.smithi093 (mon.0) 388 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2021-11-20T08:55:31.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: audit 2021-11-20T08:55:31.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: 2021-11-20T08 2021-11-20T08:55:31.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: :55:30.539513+0000 mon.smithi093 (mon.0) 389 : 2021-11-20T08:55:31.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:31.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:55:31.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: -20T08:55:30.542816+0000 mon.smithi093 ( 2021-11-20T08:55:31.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: mon.0) 390 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:31.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: cluster 2021-11 2021-11-20T08:55:31.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: -20T08:55:31.508066+0000 mgr.smithi093.qdikkm 2021-11-20T08:55:31.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: (mgr.14162) 133 : cluster [DBG] pgmap v100: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T08:55:31.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:55:31.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: -11-20T08:55:31.537499+0000 2021-11-20T08:55:31.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: mon.smithi093 (mon.0) 391 : cluster [INF] osd.4 [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] boot 2021-11-20T08:55:31.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: 2021-11-20T08:55:31.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: cluster 2021-11-20T08:55:31.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: 2021-11-20T08:55:31.537544+0000 mon.smithi093 ( 2021-11-20T08:55:31.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: mon.0) 392 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-11-20T08:55:31.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:55:31.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: -20T08:55:31.537703+0000 mon.smithi093 ( 2021-11-20T08:55:31.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:31 smithi093 conmon[33522]: mon.0) 393 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T08:55:33.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: cluster 2021-11-20T08:55:30.250348+0000 osd.4 (osd.4) 2021-11-20T08:55:33.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:55:33.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: cluster 2021-11-20T08:55:30.250470+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:33.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:32.211562+0000 mgr.smithi093.qdikkm (mgr.14162) 134 : cephadm [INF] Detected new or changed devices on smithi102 2021-11-20T08:55:33.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:32.216519+0000 mon.smithi093 (mon.0) 394 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:33.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:32.220212+0000 mon.smithi093 (mon.0) 395 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:33.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:32.223159+0000 mon.smithi093 (mon.0) 396 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:33.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: cluster 2021-11-20T08:55:32.547687+0000 mon.smithi093 (mon.0) 397 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-11-20T08:55:33.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:32.746219+0000 mon.smithi102 (mon.1) 10 : audit [INF] from='client.? 172.21.15.102:0/4222720843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fab60808-4ace-4f26-9c42-f72340134a2f"}]: dispatch 2021-11-20T08:55:33.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:32.746730+0000 mon.smithi093 (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fab60808-4ace-4f26-9c42-f72340134a2f"}]: dispatch 2021-11-20T08:55:33.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:32.750515+0000 mon.smithi093 (mon.0) 399 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fab60808-4ace-4f26-9c42-f72340134a2f"}]': finished 2021-11-20T08:55:33.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: cluster 2021-11-20T08:55:32.750560+0000 mon.smithi093 (mon.0) 400 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-20T08:55:33.423 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:32.750652+0000 mon.smithi093 (mon.0) 401 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:33.423 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:33 smithi102 conmon[36472]: audit 2021-11-20T08:55:33.151582+0000 mon.smithi102 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.102:0/2503203059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:55:33.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: cluster 2021-11-20T08:55:30.250348+0000 osd.4 (osd 2021-11-20T08:55:33.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: .4) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:55:33.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: cluster 2021-11-20T08:55:30.250470+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:33.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: cephadm 2021-11-20T08:55:32.211562+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:55:33.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 14162) 134 : cephadm [INF] Detected new or changed devices on smithi102 2021-11-20T08:55:33.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:55:33.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 20T08:55:32.216519+0000 mon.smithi093 ( 2021-11-20T08:55:33.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: mon.0) 394 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:33.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021 2021-11-20T08:55:33.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: -11-20T08:55:32.220212+0000 2021-11-20T08:55:33.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: mon.smithi093 (mon.0) 395 : audit [INF] 2021-11-20T08:55:33.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:33.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021-11-20T08:55:33.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 2021-11-20T08:55:32.223159+0000 mon.smithi093 ( 2021-11-20T08:55:33.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: mon.0) 396 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:33.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: cluster 2021-11 2021-11-20T08:55:33.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: -20T08:55:32.547687 2021-11-20T08:55:33.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 397 2021-11-20T08:55:33.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-11-20T08:55:33.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:55:33.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: -20T08:55:32.746219+0000 mon.smithi102 2021-11-20T08:55:33.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: (mon.1) 10 : audit [INF] from='client.? 172.21.15.102:0/4222720843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fab60808-4ace-4f26-9c42-f72340134a2f"}]: dispatch 2021-11-20T08:55:33.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:55:33.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: -20T08:55:32.746730+0000 2021-11-20T08:55:33.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: mon.smithi093 (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fab60808-4ace-4f26-9c42-f72340134a2f"}]: dispatch 2021-11-20T08:55:33.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 2021-11-20T08:55:33.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021-11-20T08:55:33.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 2021-11-20T08:55:32.750515+0000 mon.smithi093 ( 2021-11-20T08:55:33.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: mon.0) 399 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fab60808-4ace-4f26-9c42-f72340134a2f"}]': finished 2021-11-20T08:55:33.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:55:33.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 11-20T08:55:32.750560+0000 2021-11-20T08:55:33.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: mon.smithi093 (mon.0) 400 : cluster [DBG] 2021-11-20T08:55:33.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: osdmap e34: 6 total, 5 up, 6 in 2021-11-20T08:55:33.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021-11-20T08:55:33.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 2021-11-20T08:55:32.750652+0000 mon.smithi093 (mon 2021-11-20T08:55:33.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: .0) 401 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:33.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:55:33.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: 55:33.151582+0000 mon.smithi102 (mon.1 2021-11-20T08:55:33.720 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:33 smithi093 conmon[33522]: ) 11 : audit [DBG] from='client.? 172.21.15.102:0/2503203059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T08:55:34.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:34 smithi102 conmon[36472]: cluster 2021-11-20T08:55:33.508531+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:55:34.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:34 smithi102 conmon[36472]: .14162) 135 : cluster [DBG] pgmap v104: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:34.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:34 smithi093 conmon[33522]: cluster 2021-11-20T08:55:33 2021-11-20T08:55:34.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:34 smithi093 conmon[33522]: .508531+0000 mgr.smithi093.qdikkm (mgr.14162) 135 : cluster [DBG] pgmap v104: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:35.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:35 smithi102 conmon[36472]: cluster 2021-11-20T08:55:35.508844+0000 mgr.smithi093.qdikkm (mgr.14162) 136 : cluster [DBG] pgmap v105: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:35.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:35 smithi102 conmon[36472]: 2021-11-20T08:55:35.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:35 smithi093 conmon[33522]: cluster 2021-11-20T08:55:35.508844+0000 mgr.smithi093.qdikkm (mgr.14162) 136 : cluster [DBG] pgmap v105: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:38.123 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:37 smithi102 conmon[36472]: audit 2021-11-20T08:55:36.827647+0000 mon.smithi093 (mon.0) 402 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:38.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:37 smithi102 conmon[36472]: audit 2021-11-20T08:55:36.831009+0000 mon.smithi093 (mon.0) 403 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:38.124 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:37 smithi102 conmon[36472]: cluster 2021-11-20T08:55:37.509322+0000 mgr.smithi093.qdikkm (mgr.14162) 137 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:38.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:37 smithi093 conmon[33522]: audit 2021-11-20T08:55:36.827647+0000 mon.smithi093 (mon.0) 402 : audit 2021-11-20T08:55:38.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:37 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:38.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:37 smithi093 conmon[33522]: audit 2021-11-20T08:55:36.831009+0000 mon.smithi093 (mon.0) 403 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:38.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:37 smithi093 conmon[33522]: cluster 2021-11-20T08:55:37.509322+0000 2021-11-20T08:55:38.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:37 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 137 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:39.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:39 smithi102 conmon[36472]: audit 2021-11-20T08:55:39.204021+0000 mon.smithi093 (mon.0) 404 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T08:55:39.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:39 smithi102 conmon[36472]: audit 2021-11-20T08:55:39.204896+0000 mon.smithi093 (mon.0) 405 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:39.700 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:39 smithi093 conmon[33522]: audit 2021-11-20T08:55:39.204021+0000 mon.smithi093 (mon.0) 404 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T08:55:39.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:39 smithi093 conmon[33522]: audit 2021-11-20T08:55:39.204896+0000 mon.smithi093 (mon.0) 405 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:55:40.355 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:40 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:39.205590+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:55:40.356 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:40 smithi102 conmon[36472]: 14162) 138 : cephadm [INF] Deploying daemon osd.5 on smithi102 2021-11-20T08:55:40.356 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:40 smithi102 conmon[36472]: cluster 2021-11-20T08:55:39.509742+0000 mgr.smithi093.qdikkm (mgr.14162) 139 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:40.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:40 smithi093 conmon[33522]: cephadm 2021-11-20T08:55:39.205590+0000 mgr.smithi093.qdikkm (mgr.14162) 138 : cephadm 2021-11-20T08:55:40.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:40 smithi093 conmon[33522]: [INF] Deploying daemon osd.5 on smithi102 2021-11-20T08:55:40.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:40 smithi093 conmon[33522]: cluster 2021-11-20T08:55:39.509742+0000 mgr.smithi093.qdikkm (mgr.14162) 139 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:42.407 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.405+0000 7fcb337fe700 1 -- 172.21.15.102:0/3257782380 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcb4c057f10 con 0x7fcb3405c300 2021-11-20T08:55:42.407 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 5 on host 'smithi102' 2021-11-20T08:55:42.408 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcb3405c300 msgr2=0x7fcb3405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:42.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcb3405c300 0x7fcb3405e7c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcb40000c00 tx=0x7fcb40009b30).stop 2021-11-20T08:55:42.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f4800 msgr2=0x7fcb4c0f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:42.410 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f4800 0x7fcb4c0f73b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fcb3c002700 tx=0x7fcb3c009d80).stop 2021-11-20T08:55:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 shutdown_connections 2021-11-20T08:55:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcb3405c300 0x7fcb3405e7c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0f4800 0x7fcb4c0f73b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:42.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 --2- 172.21.15.102:0/3257782380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb4c0f5830 0x7fcb4c0f5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:42.412 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 >> 172.21.15.102:0/3257782380 conn(0x7fcb4c0f0320 msgr2=0x7fcb4c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:42.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 shutdown_connections 2021-11-20T08:55:42.413 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:42.409+0000 7fcb5090f700 1 -- 172.21.15.102:0/3257782380 wait complete. 2021-11-20T08:55:42.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:42 smithi102 conmon[36472]: cluster 2021-11-20T08:55:41.510325+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:55:42.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:42 smithi102 conmon[36472]: 140 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:42.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:42 smithi102 conmon[36472]: audit 2021-11-20T08:55:42.394939+0000 mon.smithi093 (mon.0) 406 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:42.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:42 smithi102 conmon[36472]: audit 2021-11-20T08:55:42.399811+0000 mon.smithi093 (mon.0) 407 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:42.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:42 smithi102 conmon[36472]: audit 2021-11-20T08:55:42.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:42 smithi102 conmon[36472]: 2021-11-20T08:55:42.406402+0000 mon.smithi093 (mon.0) 408 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:42.801 DEBUG:teuthology.orchestra.run.smithi102:osd.5> sudo journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.5.service 2021-11-20T08:55:42.804 INFO:tasks.cephadm:Setting up client nodes... 2021-11-20T08:55:42.805 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T08:55:42.832 INFO:journalctl@ceph.osd.5.smithi102.stdout:-- Logs begin at Sat 2021-11-20 08:41:07 UTC. -- 2021-11-20T08:55:42.840 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: cluster 2021-11-20T08:55:41.510325+0000 mgr.smithi093.qdikkm (mgr.14162) 140 : 2021-11-20T08:55:42.840 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:42.840 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: audit 2021-11-20T08:55:42.394939+0000 mon.smithi093 (mon.0) 406 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:42.840 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: audit 2021-11-20T08:55:42.399811+0000 mon.smithi093 (mon.0) 407 : audit [INF] 2021-11-20T08:55:42.841 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:42.841 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: audit 2021-11 2021-11-20T08:55:42.841 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: -20T08:55:42.406402+0000 mon.smithi093 (mon.0) 408 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:42.841 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:42 smithi093 conmon[33522]: 2021-11-20T08:55:43.653 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:43 smithi093 conmon[33522]: cluster 2021-11-20T08:55:43.510614+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:55:43.653 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:43 smithi093 conmon[33522]: 14162) 141 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:43.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:43 smithi102 conmon[36472]: cluster 2021-11-20T08:55:43.510614+0000 mgr.smithi093.qdikkm (mgr.14162) 141 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:44.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 -- 172.21.15.93:0/3923504071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f6530 msgr2=0x7f87b40f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:44.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 --2- 172.21.15.93:0/3923504071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f6530 0x7f87b40f6950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f87a4004660 tx=0x7f87a4009b30).stop 2021-11-20T08:55:44.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 -- 172.21.15.93:0/3923504071 shutdown_connections 2021-11-20T08:55:44.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 --2- 172.21.15.93:0/3923504071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b40f77d0 0x7f87b40f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:44.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 --2- 172.21.15.93:0/3923504071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f6530 0x7f87b40f6950 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:44.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 -- 172.21.15.93:0/3923504071 >> 172.21.15.93:0/3923504071 conn(0x7f87b40f1c90 msgr2=0x7f87b40f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:44.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 -- 172.21.15.93:0/3923504071 shutdown_connections 2021-11-20T08:55:44.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 -- 172.21.15.93:0/3923504071 wait complete. 2021-11-20T08:55:44.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 Processor -- start 2021-11-20T08:55:44.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.379+0000 7f87b8881700 1 -- start start 2021-11-20T08:55:44.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b8881700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b40f6530 0x7f87b406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:44.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b8881700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f77d0 0x7f87b4069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:44.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b8881700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b406a380 con 0x7f87b40f77d0 2021-11-20T08:55:44.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b8881700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b406a4c0 con 0x7f87b40f6530 2021-11-20T08:55:44.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87aaffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f77d0 0x7f87b4069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:44.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87aaffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f77d0 0x7f87b4069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42746/0 (socket says 172.21.15.93:42746) 2021-11-20T08:55:44.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87aaffd700 1 -- 172.21.15.93:0/885283543 learned_addr learned my addr 172.21.15.93:0/885283543 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:55:44.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b37fe700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b40f6530 0x7f87b406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:44.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87aaffd700 1 -- 172.21.15.93:0/885283543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b40f6530 msgr2=0x7f87b406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:44.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87aaffd700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b40f6530 0x7f87b406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:44.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87aaffd700 1 -- 172.21.15.93:0/885283543 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a4005040 con 0x7f87b40f77d0 2021-11-20T08:55:44.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87aaffd700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f77d0 0x7f87b4069e20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f879c002700 tx=0x7f879c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:44.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b0ff9700 1 -- 172.21.15.93:0/885283543 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c009320 con 0x7f87b40f77d0 2021-11-20T08:55:44.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b0ff9700 1 -- 172.21.15.93:0/885283543 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f879c00f040 con 0x7f87b40f77d0 2021-11-20T08:55:44.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b406a7a0 con 0x7f87b40f77d0 2021-11-20T08:55:44.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.380+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b410ed70 con 0x7f87b40f77d0 2021-11-20T08:55:44.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.381+0000 7f87b0ff9700 1 -- 172.21.15.93:0/885283543 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c014870 con 0x7f87b40f77d0 2021-11-20T08:55:44.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.381+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8798004fa0 con 0x7f87b40f77d0 2021-11-20T08:55:44.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.382+0000 7f87b0ff9700 1 -- 172.21.15.93:0/885283543 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f879c0149d0 con 0x7f87b40f77d0 2021-11-20T08:55:44.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.382+0000 7f87b0ff9700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f87a005c300 0x7f87a005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:44.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.382+0000 7f87b0ff9700 1 -- 172.21.15.93:0/885283543 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f879c075c00 con 0x7f87b40f77d0 2021-11-20T08:55:44.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.382+0000 7f87b37fe700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f87a005c300 0x7f87a005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:44.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.382+0000 7f87b37fe700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f87a005c300 0x7f87a005e7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f87b406b730 tx=0x7f87a4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:44.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.386+0000 7f87b0ff9700 1 -- 172.21.15.93:0/885283543 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f879c07b020 con 0x7f87b40f77d0 2021-11-20T08:55:44.601 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.602+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f8798005670 con 0x7f87b40f77d0 2021-11-20T08:55:44.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.606+0000 7f87b0ff9700 1 -- 172.21.15.93:0/885283543 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f879c049e00 con 0x7f87b40f77d0 2021-11-20T08:55:44.606 INFO:teuthology.orchestra.run.smithi093.stdout:[client.0] 2021-11-20T08:55:44.607 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQAQuJhhamD7IxAAv6K1r8cKwdgnRZCm4jmKDw== 2021-11-20T08:55:44.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f87a005c300 msgr2=0x7f87a005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:44.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f87a005c300 0x7f87a005e7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f87b406b730 tx=0x7f87a4009b30).stop 2021-11-20T08:55:44.609 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f77d0 msgr2=0x7f87b4069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:44.609 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f77d0 0x7f87b4069e20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f879c002700 tx=0x7f879c009e00).stop 2021-11-20T08:55:44.610 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 shutdown_connections 2021-11-20T08:55:44.610 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f87a005c300 0x7f87a005e7c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:44.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87b40f6530 0x7f87b406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:44.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 --2- 172.21.15.93:0/885283543 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87b40f77d0 0x7f87b4069e20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:44.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.607+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 >> 172.21.15.93:0/885283543 conn(0x7f87b40f1c90 msgr2=0x7f87b40fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:44.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.608+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 shutdown_connections 2021-11-20T08:55:44.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:44.608+0000 7f87b8881700 1 -- 172.21.15.93:0/885283543 wait complete. 2021-11-20T08:55:44.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:44 smithi102 conmon[36472]: audit 2021-11-20T08:55:44.603522+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:44.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:44 smithi102 conmon[36472]: 409 : audit [INF] from='client.? 172.21.15.93:0/885283543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T08:55:44.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:44 smithi102 conmon[36472]: audit 2021-11-20T08:55:44.606855+0000 mon.smithi093 (mon.0) 410 : audit [INF] from='client.? 172.21.15.93:0/885283543' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T08:55:44.922 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:44 smithi093 conmon[33522]: audit 2021-11-20T08:55:44.603522+0000 mon.smithi093 (mon.0) 409 : audit [INF] from='client.? 172.21.15.93:0/885283543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T08:55:44.923 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:44 smithi093 conmon[33522]: audit 2021-11-20T08:55:44.606855+0000 mon.smithi093 (mon.0) 410 : audit [INF] from='client.? 172.21.15.93:0/885283543' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T08:55:44.978 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2021-11-20T08:55:44.979 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-11-20T08:55:44.980 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-11-20T08:55:45.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T08:55:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:46 smithi102 conmon[36472]: audit 2021-11-20T08:55:45.136381+0000 mon.smithi093 (mon.0) 411 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:46.317 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:46 smithi102 conmon[36472]: cluster 2021-11-20T08:55:45.511003+0000 mgr.smithi093.qdikkm (mgr.14162) 142 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:46 smithi093 conmon[33522]: audit 2021-11-20T08:55:45.136381+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:46 smithi093 conmon[33522]: 411 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:46 smithi093 conmon[33522]: cluster 2021-11-20T08:55:45.511003+0000 mgr.smithi093.qdikkm (mgr.14162) 142 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:46.668 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 08:55:46 smithi102 conmon[52795]: debug 2021-11-20T08:55:46.317+0000 7f8e501e3080 -1 osd.5 0 log_to_monitors {default=true} 2021-11-20T08:55:46.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 -- 172.21.15.102:0/1841902614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 msgr2=0x7f32d40f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:46.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 --2- 172.21.15.102:0/1841902614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d40f6950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f32c4004660 tx=0x7f32c4009b30).stop 2021-11-20T08:55:46.956 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 -- 172.21.15.102:0/1841902614 shutdown_connections 2021-11-20T08:55:46.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 --2- 172.21.15.102:0/1841902614 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32d40f77d0 0x7f32d40f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:46.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 --2- 172.21.15.102:0/1841902614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d40f6950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:46.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 -- 172.21.15.102:0/1841902614 >> 172.21.15.102:0/1841902614 conn(0x7f32d40f1c90 msgr2=0x7f32d40f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:46.957 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 -- 172.21.15.102:0/1841902614 shutdown_connections 2021-11-20T08:55:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 -- 172.21.15.102:0/1841902614 wait complete. 2021-11-20T08:55:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.953+0000 7f32d9478700 1 Processor -- start 2021-11-20T08:55:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d9478700 1 -- start start 2021-11-20T08:55:46.958 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d9478700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d4106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:46.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d9478700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32d40f77d0 0x7f32d4107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:46.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d9478700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4107880 con 0x7f32d40f77d0 2021-11-20T08:55:46.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d9478700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d41080c0 con 0x7f32d40f6530 2021-11-20T08:55:46.959 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d4106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:46.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d4106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53876/0 (socket says 172.21.15.102:53876) 2021-11-20T08:55:46.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d3fff700 1 -- 172.21.15.102:0/3895764406 learned_addr learned my addr 172.21.15.102:0/3895764406 (peer_addr_for_me v2:172.21.15.102:0/0) 2021-11-20T08:55:46.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d37fe700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32d40f77d0 0x7f32d4107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:46.960 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d3fff700 1 -- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32d40f77d0 msgr2=0x7f32d4107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:46.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d3fff700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32d40f77d0 0x7f32d4107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:46.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d3fff700 1 -- 172.21.15.102:0/3895764406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c4005040 con 0x7f32d40f6530 2021-11-20T08:55:46.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.954+0000 7f32d3fff700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d4106d90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f32c4000c00 tx=0x7f32c400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:46.961 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.955+0000 7f32d0ff9700 1 -- 172.21.15.102:0/3895764406 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c4019070 con 0x7f32d40f6530 2021-11-20T08:55:46.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.955+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d4108340 con 0x7f32d40f6530 2021-11-20T08:55:46.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.955+0000 7f32d0ff9700 1 -- 172.21.15.102:0/3895764406 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32c4007e70 con 0x7f32d40f6530 2021-11-20T08:55:46.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.955+0000 7f32d0ff9700 1 -- 172.21.15.102:0/3895764406 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c4014d40 con 0x7f32d40f6530 2021-11-20T08:55:46.962 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.955+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d4108820 con 0x7f32d40f6530 2021-11-20T08:55:46.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.956+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d4059100 con 0x7f32d40f6530 2021-11-20T08:55:46.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.956+0000 7f32d0ff9700 1 -- 172.21.15.102:0/3895764406 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f32c4025070 con 0x7f32d40f6530 2021-11-20T08:55:46.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.956+0000 7f32d0ff9700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32bc05c300 0x7f32bc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:46.963 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.956+0000 7f32d0ff9700 1 -- 172.21.15.102:0/3895764406 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f32c40769d0 con 0x7f32d40f6530 2021-11-20T08:55:46.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.956+0000 7f32d37fe700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32bc05c300 0x7f32bc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:46.964 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.957+0000 7f32d37fe700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32bc05c300 0x7f32bc05e7c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f32d4106b20 tx=0x7f32c800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:46.965 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:46.960+0000 7f32d0ff9700 1 -- 172.21.15.102:0/3895764406 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32c4049b10 con 0x7f32d40f6530 2021-11-20T08:55:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.179+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f32d40fed60 con 0x7f32d40f6530 2021-11-20T08:55:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.183+0000 7f32d0ff9700 1 -- 172.21.15.102:0/3895764406 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f32c404d130 con 0x7f32d40f6530 2021-11-20T08:55:47.183 INFO:teuthology.orchestra.run.smithi102.stdout:[client.1] 2021-11-20T08:55:47.183 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQATuJhhWUHPChAAqWo+uToPB8Bnfqm2vh+8lQ== 2021-11-20T08:55:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32bc05c300 msgr2=0x7f32bc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32bc05c300 0x7f32bc05e7c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f32d4106b20 tx=0x7f32c800b040).stop 2021-11-20T08:55:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 msgr2=0x7f32d4106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:47.185 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d4106d90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f32c4000c00 tx=0x7f32c400b040).stop 2021-11-20T08:55:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 shutdown_connections 2021-11-20T08:55:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32bc05c300 0x7f32bc05e7c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32d40f6530 0x7f32d4106d90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:47.186 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 --2- 172.21.15.102:0/3895764406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32d40f77d0 0x7f32d4107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 >> 172.21.15.102:0/3895764406 conn(0x7f32d40f1c90 msgr2=0x7f32d40fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 shutdown_connections 2021-11-20T08:55:47.187 INFO:teuthology.orchestra.run.smithi102.stderr:2021-11-20T08:55:47.185+0000 7f32d9478700 1 -- 172.21.15.102:0/3895764406 wait complete. 2021-11-20T08:55:47.212 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:47 smithi102 conmon[36472]: audit 2021-11-20T08:55:46.321558+0000 mon.smithi102 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T08:55:47.213 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:47 smithi102 conmon[36472]: audit 2021-11-20T08:55:46.321897+0000 mon.smithi093 (mon.0) 412 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T08:55:47.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:47 smithi093 conmon[33522]: audit 2021-11-20T08:55:46.321558+0000 mon.smithi102 ( 2021-11-20T08:55:47.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:47 smithi093 conmon[33522]: mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T08:55:47.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:47 smithi093 conmon[33522]: audit 2021-11-20T08:55:46.321897+0000 mon.smithi093 (mon.0) 412 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T08:55:47.572 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2021-11-20T08:55:47.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-11-20T08:55:47.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-11-20T08:55:47.616 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T08:55:48.047 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:55:48.250 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.146060 2021-11-20T08:55:48.251 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 413 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T08:55:48.251 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: cluster 2021-11-20T08:55:47.146132+0000 mon.smithi093 (mon.0) 414 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-11-20T08:55:48.251 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.146552+0000 mon.smithi093 (mon.0) 415 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:48.251 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.146984+0000 mon.smithi102 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:48.252 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.149879+0000 mon.smithi093 (mon.0) 416 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:48.252 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.180856+0000 mon.smithi102 (mon.1) 14 : audit [INF] from='client.? 172.21.15.102:0/3895764406' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T08:55:48.252 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.181222+0000 mon.smithi093 (mon.0) 417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T08:55:48.252 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.183941+0000 mon.smithi093 (mon.0) 418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T08:55:48.253 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: audit 2021-11-20T08:55:47.404736+0000 mon.smithi093 (mon.0) 419 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:48.253 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[36472]: cluster 2021-11-20T08:55:47.511223+0000 mgr.smithi093.qdikkm (mgr.14162) 143 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:48.254 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[52795]: debug 2021-11-20T08:55:48.152+0000 7f8e46754700 -1 osd.5 0 waiting for initial osdmap 2021-11-20T08:55:48.254 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 08:55:48 smithi102 conmon[52795]: debug 2021-11-20T08:55:48.162+0000 7f8e416cd700 -1 osd.5 36 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T08:55:48.411 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.146060+0000 mon.smithi093 ( 2021-11-20T08:55:48.411 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: mon.0) 413 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T08:55:48.412 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: cluster 2021-11-20T08:55:47.146132+0000 mon.smithi093 (mon.0) 414 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-11-20T08:55:48.412 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.146552+0000 mon.smithi093 (mon.0) 415 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:48.412 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.146984+0000 mon.smithi102 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:48.412 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.149879+0000 mon.smithi093 (mon.0) 416 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2021-11-20T08:55:48.413 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.180856+0000 mon.smithi102 (mon.1) 14 : audit [INF] from='client.? 172.21.15.102:0/3895764406' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T08:55:48.413 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.181222+0000 mon.smithi093 (mon.0) 417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T08:55:48.413 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.183941+0000 mon.smithi093 (mon.0) 418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T08:55:48.414 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: audit 2021-11-20T08:55:47.404736+0000 mon.smithi093 (mon.0) 419 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:48.414 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T08:55:48.414 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:48 smithi093 conmon[33522]: T08:55:47.511223+0000 mgr.smithi093.qdikkm (mgr.14162) 143 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:49.238 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 -- 172.21.15.93:0/680103406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f56c0 msgr2=0x7f56b00f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:49.239 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 --2- 172.21.15.93:0/680103406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f56c0 0x7f56b00f5ae0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f56a0004660 tx=0x7f56a0009b30).stop 2021-11-20T08:55:49.239 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 -- 172.21.15.93:0/680103406 shutdown_connections 2021-11-20T08:55:49.239 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 --2- 172.21.15.93:0/680103406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56b00f6960 0x7f56b00f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:49.239 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 --2- 172.21.15.93:0/680103406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f56c0 0x7f56b00f5ae0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:49.240 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 -- 172.21.15.93:0/680103406 >> 172.21.15.93:0/680103406 conn(0x7f56b00f0e20 msgr2=0x7f56b00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:49.240 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 -- 172.21.15.93:0/680103406 shutdown_connections 2021-11-20T08:55:49.240 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.236+0000 7f56b5882700 1 -- 172.21.15.93:0/680103406 wait complete. 2021-11-20T08:55:49.240 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56b5882700 1 Processor -- start 2021-11-20T08:55:49.241 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56b5882700 1 -- start start 2021-11-20T08:55:49.241 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56b5882700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56b00f56c0 0x7f56b0109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:49.241 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56b5882700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f6960 0x7f56b010c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:49.241 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56b5882700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b0060940 con 0x7f56b00f6960 2021-11-20T08:55:49.241 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56b5882700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b0060a80 con 0x7f56b00f56c0 2021-11-20T08:55:49.242 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56ae7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f6960 0x7f56b010c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:49.242 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56ae7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f6960 0x7f56b010c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42760/0 (socket says 172.21.15.93:42760) 2021-11-20T08:55:49.242 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56ae7fc700 1 -- 172.21.15.93:0/3041097969 learned_addr learned my addr 172.21.15.93:0/3041097969 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:55:49.242 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56aeffd700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56b00f56c0 0x7f56b0109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:49.243 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56ae7fc700 1 -- 172.21.15.93:0/3041097969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56b00f56c0 msgr2=0x7f56b0109dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:49.243 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56ae7fc700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56b00f56c0 0x7f56b0109dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:49.243 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56ae7fc700 1 -- 172.21.15.93:0/3041097969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a0005040 con 0x7f56b00f6960 2021-11-20T08:55:49.243 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.237+0000 7f56ae7fc700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f6960 0x7f56b010c2d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f569800c530 tx=0x7f5698007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:49.243 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.238+0000 7f56a77fe700 1 -- 172.21.15.93:0/3041097969 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f569800f650 con 0x7f56b00f6960 2021-11-20T08:55:49.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.238+0000 7f56a77fe700 1 -- 172.21.15.93:0/3041097969 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f569800fc60 con 0x7f56b00f6960 2021-11-20T08:55:49.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.238+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b010c880 con 0x7f56b00f6960 2021-11-20T08:55:49.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.238+0000 7f56a77fe700 1 -- 172.21.15.93:0/3041097969 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5698012c30 con 0x7f56b00f6960 2021-11-20T08:55:49.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.238+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b010ce40 con 0x7f56b00f6960 2021-11-20T08:55:49.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.239+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b0045ad0 con 0x7f56b00f6960 2021-11-20T08:55:49.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.239+0000 7f56a77fe700 1 -- 172.21.15.93:0/3041097969 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5698010070 con 0x7f56b00f6960 2021-11-20T08:55:49.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.239+0000 7f56a77fe700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f569c05c2a0 0x7f569c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:49.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.239+0000 7f56a77fe700 1 -- 172.21.15.93:0/3041097969 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5698075750 con 0x7f56b00f6960 2021-11-20T08:55:49.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.239+0000 7f56aeffd700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f569c05c2a0 0x7f569c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:49.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.240+0000 7f56aeffd700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f569c05c2a0 0x7f569c05e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f56b00605c0 tx=0x7f56a0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:49.247 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.243+0000 7f56a77fe700 1 -- 172.21.15.93:0/3041097969 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56980452b0 con 0x7f56b00f6960 2021-11-20T08:55:49.401 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.402+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56b0027a00 con 0x7f56b00f6960 2021-11-20T08:55:49.402 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:49 smithi093 conmon[33522]: audit 2021-11-20T08:55:48.150454+0000 mon.smithi093 (mon.0) 420 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2021-11-20T08:55:49.403 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:49 smithi093 conmon[33522]: cluster 2021-11-20T08:55:48.150534+0000 mon.smithi093 (mon.0) 421 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2021-11-20T08:55:49.403 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:49 smithi093 conmon[33522]: audit 2021-11-20T08:55:48.150840+0000 mon.smithi093 (mon.0) 422 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:49.404 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:49 smithi093 conmon[33522]: audit 2021-11-20T08:55:48.154311+0000 mon.smithi093 (mon.0) 423 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:49.404 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:49 smithi093 conmon[33522]: audit 2021-11-20T08:55:48.643044+0000 mon.smithi093 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:49.405 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:49 smithi093 conmon[33522]: audit 2021-11-20T08:55:48.987760+0000 mon.smithi093 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:49.405 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:49 smithi093 conmon[33522]: audit 2021-11-20T08:55:48.990310+0000 mon.smithi093 (mon.0) 426 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:49.406 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.403+0000 7f56a77fe700 1 -- 172.21.15.93:0/3041097969 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+10888 (secure 0 0 0) 0x7f56980487d0 con 0x7f56b00f6960 2021-11-20T08:55:49.407 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:55:49.407 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":37,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","created":"2021-11-20T08:51:57.464444+0000","modified":"2021-11-20T08:55:49.151657+0000","last_up_change":"2021-11-20T08:55:49.151657+0000","last_in_change":"2021-11-20T08:55:32.747237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T08:55:09.134409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b62eaab5-818e-4739-98c3-d386425d4837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6803","nonce":1548414389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6805","nonce":1548414389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6809","nonce":1548414389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6807","nonce":1548414389}]},"public_addr":"172.21.15.93:6803/1548414389","cluster_addr":"172.21.15.93:6805/1548414389","heartbeat_back_addr":"172.21.15.93:6809/1548414389","heartbeat_front_addr":"172.21.15.93:6807/1548414389","state":["exists","up"]},{"osd":1,"uuid":"a46f3336-3bc4-4736-9095-c4d72e662323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6811","nonce":3785015356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6813","nonce":3785015356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6817","nonce":3785015356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6815","nonce":3785015356}]},"public_addr":"172.21.15.93:6811/3785015356","cluster_addr":"172.21.15.93:6813/3785015356","heartbeat_back_addr":"172.21.15.93:6817/3785015356","heartbeat_front_addr":"172.21.15.93:6815/3785015356","state":["exists","up"]},{"osd":2,"uuid":"72096543-af61-429d-a8d1-a77726a1f3eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6819","nonce":82486730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6821","nonce":82486730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6825","nonce":82486730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6823","nonce":82486730}]},"public_addr":"172.21.15.93:6819/82486730","cluster_addr":"172.21.15.93:6821/82486730","heartbeat_back_addr":"172.21.15.93:6825/82486730","heartbeat_front_addr":"172.21.15.93:6823/82486730","state":["exists","up"]},{"osd":3,"uuid":"37ba4275-7023-4f74-ba1f-7ad2f864476f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6801","nonce":1093924688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6803","nonce":1093924688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6807","nonce":1093924688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6805","nonce":1093924688}]},"public_addr":"172.21.15.102:6801/1093924688","cluster_addr":"172.21.15.102:6803/1093924688","heartbeat_back_addr":"172.21.15.102:6807/1093924688","heartbeat_front_addr":"172.21.15.102:6805/1093924688","state":["exists","up"]},{"osd":4,"uuid":"c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6809","nonce":2700333748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6811","nonce":2700333748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6815","nonce":2700333748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6813","nonce":2700333748}]},"public_addr":"172.21.15.102:6809/2700333748","cluster_addr":"172.21.15.102:6811/2700333748","heartbeat_back_addr":"172.21.15.102:6815/2700333748","heartbeat_front_addr":"172.21.15.102:6813/2700333748","state":["exists","up"]},{"osd":5,"uuid":"fab60808-4ace-4f26-9c42-f72340134a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6817","nonce":4083093586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6819","nonce":4083093586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6823","nonce":4083093586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6821","nonce":4083093586}]},"public_addr":"172.21.15.102:6817/4083093586","cluster_addr":"172.21.15.102:6819/4083093586","heartbeat_back_addr":"172.21.15.102:6823/4083093586","heartbeat_front_addr":"172.21.15.102:6821/4083093586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:21.010676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:41.314338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:57.961372+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:13.733626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:30.250474+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:0/3689853676":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/3631043468":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/485709266":"2021-11-21T08:52:37.986811+0000","172.21.15.93:6800/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1880974768":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1518372611":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/2364704567":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/1580490298":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6801/1580490298":"2021-11-21T08:52:15.847551+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T08:55:49.408 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f569c05c2a0 msgr2=0x7f569c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:49.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f569c05c2a0 0x7f569c05e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f56b00605c0 tx=0x7f56a0009b30).stop 2021-11-20T08:55:49.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f6960 msgr2=0x7f56b010c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:49.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f6960 0x7f56b010c2d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f569800c530 tx=0x7f5698007e20).stop 2021-11-20T08:55:49.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 shutdown_connections 2021-11-20T08:55:49.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f569c05c2a0 0x7f569c05e760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:49.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56b00f56c0 0x7f56b0109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:49.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 --2- 172.21.15.93:0/3041097969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56b00f6960 0x7f56b010c2d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:49.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.404+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 >> 172.21.15.93:0/3041097969 conn(0x7f56b00f0e20 msgr2=0x7f56b00fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:49.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.405+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 shutdown_connections 2021-11-20T08:55:49.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:49.405+0000 7f56b5882700 1 -- 172.21.15.93:0/3041097969 wait complete. 2021-11-20T08:55:49.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:49 smithi102 conmon[36472]: audit 2021-11-20T08:55:48.150454+0000 mon.smithi093 (mon.0) 420 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2021-11-20T08:55:49.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:49 smithi102 conmon[36472]: cluster 2021-11-20T08:55:48.150534+0000 mon.smithi093 (mon.0) 421 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2021-11-20T08:55:49.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:49 smithi102 conmon[36472]: audit 2021-11-20T08:55:48.150840+0000 mon.smithi093 (mon.0) 422 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:49.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:49 smithi102 conmon[36472]: audit 2021-11-20T08:55:48.154311+0000 mon.smithi093 (mon.0) 423 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:49.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:49 smithi102 conmon[36472]: audit 2021-11-20T08:55:48.643044+0000 mon.smithi093 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:49.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:49 smithi102 conmon[36472]: audit 2021-11-20T08:55:48.987760+0000 mon.smithi093 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:49.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:49 smithi102 conmon[36472]: audit 2021-11-20T08:55:48.990310+0000 mon.smithi093 (mon.0) 426 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:55:49.747 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T08:55:09.134409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2021-11-20T08:55:49.749 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-11-20T08:55:50.290 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:55:50.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: cluster 2021-11-20T08:55:47.331429+0000 osd.5 (osd.5) 1 : cluster 2021-11-20T08:55:50.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: [DBG] purged_snaps scrub starts 2021-11-20T08:55:50.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: cluster 2021-11-20T08:55:47.331555+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:50.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: cephadm 2021-11-20T08:55:48 2021-11-20T08:55:50.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: .638254+0000 mgr.smithi093.qdikkm (mgr.14162) 144 : cephadm [INF] Detected new or changed devices on smithi102 2021-11-20T08:55:50.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:55:50.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: -11-20T08:55:49.155305+0000 mon.smithi093 (mon.0) 427 : cluster 2021-11-20T08:55:50.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: [INF] osd.5 [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] boot 2021-11-20T08:55:50.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: cluster 2021-11-20T08:55:49.155342+0000 mon.smithi093 2021-11-20T08:55:50.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: (mon.0) 428 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T08:55:50.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: audit 2021-11-20T08:55:49.155730+0000 mon.smithi093 2021-11-20T08:55:50.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: (mon.0) 429 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:50.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: audit 2021-11-20T08:55:49.403481+0000 mon.smithi093 ( 2021-11-20T08:55:50.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: mon.0) 430 : audit [DBG] from='client.? 172.21.15.93:0/3041097969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:55:50.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T08:55:50.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: -20T08:55:49.511566+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:55:50.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:50 smithi102 conmon[36472]: ) 145 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:50.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: cluster 2021-11-20T08:55:47.331429+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T08:55:50.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: cluster 2021-11-20T08:55:47.331555+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T08:55:50.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: cephadm 2021-11-20T08:55:48.638254+0000 mgr.smithi093.qdikkm (mgr.14162) 144 : cephadm [INF] Detected new or changed devices on smithi102 2021-11-20T08:55:50.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: cluster 2021-11-20T08:55:49.155305+0000 mon.smithi093 (mon.0) 427 : cluster [INF] osd.5 [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] boot 2021-11-20T08:55:50.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: cluster 2021-11-20T08:55:49.155342+0000 mon.smithi093 (mon.0) 428 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T08:55:50.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: audit 2021-11-20T08:55:49.155730+0000 mon.smithi093 (mon.0) 429 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T08:55:50.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: audit 2021-11-20T08:55:49.403481+0000 mon.smithi093 (mon.0) 430 : audit [DBG] from='client.? 172.21.15.93:0/3041097969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:55:50.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:50 smithi093 conmon[33522]: cluster 2021-11-20T08:55:49.511566+0000 mgr.smithi093.qdikkm (mgr.14162) 145 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T08:55:51.210 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:51 smithi093 conmon[33522]: cluster 2021-11-20T08:55:50.164656+0000 mon.smithi093 (mon.0) 431 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T08:55:51.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:51 smithi102 conmon[36472]: cluster 2021-11-20T08:55:50.164656+0000 mon.smithi093 (mon.0) 431 : cluster 2021-11-20T08:55:51.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:51 smithi102 conmon[36472]: [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T08:55:51.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 -- 172.21.15.93:0/2973131105 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 msgr2=0x7f73900f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:51.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 --2- 172.21.15.93:0/2973131105 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f73900f6dc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f7378004660 tx=0x7f7378009b30).stop 2021-11-20T08:55:51.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 -- 172.21.15.93:0/2973131105 shutdown_connections 2021-11-20T08:55:51.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 --2- 172.21.15.93:0/2973131105 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f73900f6dc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:51.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 --2- 172.21.15.93:0/2973131105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73900f56c0 0x7f73900f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:51.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 -- 172.21.15.93:0/2973131105 >> 172.21.15.93:0/2973131105 conn(0x7f73900f0e20 msgr2=0x7f73900f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:51.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 -- 172.21.15.93:0/2973131105 shutdown_connections 2021-11-20T08:55:51.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 -- 172.21.15.93:0/2973131105 wait complete. 2021-11-20T08:55:51.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 Processor -- start 2021-11-20T08:55:51.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.559+0000 7f7396fea700 1 -- start start 2021-11-20T08:55:51.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7396fea700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73900f56c0 0x7f7390109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:51.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7396fea700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f739010c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:51.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7396fea700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7390060940 con 0x7f73900f6960 2021-11-20T08:55:51.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7396fea700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7390060a80 con 0x7f73900f56c0 2021-11-20T08:55:51.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f739010c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:51.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f739010c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42780/0 (socket says 172.21.15.93:42780) 2021-11-20T08:55:51.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738ffff700 1 -- 172.21.15.93:0/4269129282 learned_addr learned my addr 172.21.15.93:0/4269129282 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:55:51.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7394d86700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73900f56c0 0x7f7390109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:51.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738ffff700 1 -- 172.21.15.93:0/4269129282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73900f56c0 msgr2=0x7f7390109da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:51.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738ffff700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73900f56c0 0x7f7390109da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:51.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738ffff700 1 -- 172.21.15.93:0/4269129282 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7378005040 con 0x7f73900f6960 2021-11-20T08:55:51.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7394d86700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73900f56c0 0x7f7390109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:55:51.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738ffff700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f739010c2a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7378009f50 tx=0x7f7378009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:51.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738d7fa700 1 -- 172.21.15.93:0/4269129282 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7378018070 con 0x7f73900f6960 2021-11-20T08:55:51.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738d7fa700 1 -- 172.21.15.93:0/4269129282 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7378013410 con 0x7f73900f6960 2021-11-20T08:55:51.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739010c7f0 con 0x7f73900f6960 2021-11-20T08:55:51.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f738d7fa700 1 -- 172.21.15.93:0/4269129282 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737801c3c0 con 0x7f73900f6960 2021-11-20T08:55:51.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.560+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739010cd50 con 0x7f73900f6960 2021-11-20T08:55:51.595 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.562+0000 7f738d7fa700 1 -- 172.21.15.93:0/4269129282 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f737801d420 con 0x7f73900f6960 2021-11-20T08:55:51.597 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.562+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7390045ad0 con 0x7f73900f6960 2021-11-20T08:55:51.597 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.562+0000 7f738d7fa700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f737c05c2a0 0x7f737c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:51.597 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.562+0000 7f738d7fa700 1 -- 172.21.15.93:0/4269129282 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7378077860 con 0x7f73900f6960 2021-11-20T08:55:51.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.562+0000 7f7394d86700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f737c05c2a0 0x7f737c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:51.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.563+0000 7f7394d86700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f737c05c2a0 0x7f737c05e760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f73900605c0 tx=0x7f7380004d00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:51.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.566+0000 7f738d7fa700 1 -- 172.21.15.93:0/4269129282 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f737804a860 con 0x7f73900f6960 2021-11-20T08:55:51.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.732+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7390027a00 con 0x7f73900f6960 2021-11-20T08:55:51.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.733+0000 7f738d7fa700 1 -- 172.21.15.93:0/4269129282 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v38) v1 ==== 110+0+10 (secure 0 0 0) 0x7f737804de80 con 0x7f73900f6960 2021-11-20T08:55:51.733 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 1 2021-11-20T08:55:51.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f737c05c2a0 msgr2=0x7f737c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:51.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f737c05c2a0 0x7f737c05e760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f73900605c0 tx=0x7f7380004d00).stop 2021-11-20T08:55:51.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 msgr2=0x7f739010c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:51.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f739010c2a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7378009f50 tx=0x7f7378009b30).stop 2021-11-20T08:55:51.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 shutdown_connections 2021-11-20T08:55:51.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f737c05c2a0 0x7f737c05e760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:51.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73900f56c0 0x7f7390109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:51.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 --2- 172.21.15.93:0/4269129282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73900f6960 0x7f739010c2a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:51.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 >> 172.21.15.93:0/4269129282 conn(0x7f73900f0e20 msgr2=0x7f73900fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:51.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 shutdown_connections 2021-11-20T08:55:51.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:51.734+0000 7f7396fea700 1 -- 172.21.15.93:0/4269129282 wait complete. 2021-11-20T08:55:52.038 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-11-20T08:55:52.040 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-11-20T08:55:52.040 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph mgr dump --format=json 2021-11-20T08:55:52.586 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:55:52.790 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:52 smithi093 conmon[33522]: cluster 2021-11-20T08:55:51.511872+0000 mgr.smithi093.qdikkm (mgr.14162) 146 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:52.791 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:52 smithi093 conmon[33522]: audit 2021-11-20T08:55:51.733774+0000 mon.smithi093 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.93:0/4269129282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:55:52.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:52 smithi102 conmon[36472]: cluster 2021-11-20T08:55:51.511872+0000 mgr.smithi093.qdikkm (mgr.14162) 146 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:52.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:52 smithi102 conmon[36472]: audit 2021-11-20T08:55:51.733774+0000 mon.smithi093 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.93:0/4269129282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:55:53.812 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:53 smithi093 conmon[33522]: cluster 2021-11-20T08:55:53.512140+0000 mgr.smithi093.qdikkm (mgr.14162) 147 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:53.978 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:53 smithi102 conmon[36472]: cluster 2021-11-20T08:55:53.512140+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:55:53.978 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:53 smithi102 conmon[36472]: .14162) 147 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:53.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 -- 172.21.15.93:0/1799992980 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 msgr2=0x7faae80fc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:53.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 --2- 172.21.15.93:0/1799992980 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80fc070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7faadc004660 tx=0x7faadc009b30).stop 2021-11-20T08:55:53.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 -- 172.21.15.93:0/1799992980 shutdown_connections 2021-11-20T08:55:53.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 --2- 172.21.15.93:0/1799992980 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80fc070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:53.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 --2- 172.21.15.93:0/1799992980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faae80f44a0 0x7faae80f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:53.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 -- 172.21.15.93:0/1799992980 >> 172.21.15.93:0/1799992980 conn(0x7faae80f0140 msgr2=0x7faae80f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:53.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 -- 172.21.15.93:0/1799992980 shutdown_connections 2021-11-20T08:55:54.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 -- 172.21.15.93:0/1799992980 wait complete. 2021-11-20T08:55:54.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 Processor -- start 2021-11-20T08:55:54.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 -- start start 2021-11-20T08:55:54.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.994+0000 7faaf064e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faae80f44a0 0x7faae80f7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:54.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaf064e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:54.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaf064e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80f64c0 con 0x7faae80f4db0 2021-11-20T08:55:54.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaf064e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80f6600 con 0x7faae80f44a0 2021-11-20T08:55:54.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaedbe9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:54.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaedbe9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42794/0 (socket says 172.21.15.93:42794) 2021-11-20T08:55:54.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaedbe9700 1 -- 172.21.15.93:0/3125573024 learned_addr learned my addr 172.21.15.93:0/3125573024 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:55:54.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaee3ea700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faae80f44a0 0x7faae80f7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:54.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaedbe9700 1 -- 172.21.15.93:0/3125573024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faae80f44a0 msgr2=0x7faae80f7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:54.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaedbe9700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faae80f44a0 0x7faae80f7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:54.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaedbe9700 1 -- 172.21.15.93:0/3125573024 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc005040 con 0x7faae80f4db0 2021-11-20T08:55:54.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaedbe9700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80f5f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7faadc009830 tx=0x7faadc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:54.004 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faadaffd700 1 -- 172.21.15.93:0/3125573024 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faadc018070 con 0x7faae80f4db0 2021-11-20T08:55:54.004 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faadaffd700 1 -- 172.21.15.93:0/3125573024 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faadc013410 con 0x7faae80f4db0 2021-11-20T08:55:54.004 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae80f6880 con 0x7faae80f4db0 2021-11-20T08:55:54.004 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faadaffd700 1 -- 172.21.15.93:0/3125573024 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faadc01c620 con 0x7faae80f4db0 2021-11-20T08:55:54.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.995+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae80f6de0 con 0x7faae80f4db0 2021-11-20T08:55:54.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.996+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae8045ad0 con 0x7faae80f4db0 2021-11-20T08:55:54.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.996+0000 7faadaffd700 1 -- 172.21.15.93:0/3125573024 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faadc01c890 con 0x7faae80f4db0 2021-11-20T08:55:54.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.997+0000 7faadaffd700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faad405c2a0 0x7faad405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:54.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.997+0000 7faadaffd700 1 -- 172.21.15.93:0/3125573024 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7faadc077970 con 0x7faae80f4db0 2021-11-20T08:55:54.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.997+0000 7faaee3ea700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faad405c2a0 0x7faad405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:54.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:53.997+0000 7faaee3ea700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faad405c2a0 0x7faad405e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7faae4007fd0 tx=0x7faae4009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:54.007 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.001+0000 7faadaffd700 1 -- 172.21.15.93:0/3125573024 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faadc04a990 con 0x7faae80f4db0 2021-11-20T08:55:54.201 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.202+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7faae8027a00 con 0x7faae80f4db0 2021-11-20T08:55:54.204 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.205+0000 7faadaffd700 1 -- 172.21.15.93:0/3125573024 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+143498 (secure 0 0 0) 0x7faadc04dfb0 con 0x7faae80f4db0 2021-11-20T08:55:54.207 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:55:54.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faad405c2a0 msgr2=0x7faad405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:54.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faad405c2a0 0x7faad405e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7faae4007fd0 tx=0x7faae4009040).stop 2021-11-20T08:55:54.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 msgr2=0x7faae80f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:54.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80f5f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7faadc009830 tx=0x7faadc009b30).stop 2021-11-20T08:55:54.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 shutdown_connections 2021-11-20T08:55:54.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faad405c2a0 0x7faad405e760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:54.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faae80f44a0 0x7faae80f7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:54.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 --2- 172.21.15.93:0/3125573024 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faae80f4db0 0x7faae80f5f60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:54.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.207+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 >> 172.21.15.93:0/3125573024 conn(0x7faae80f0140 msgr2=0x7faae80f8870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:54.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.208+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 shutdown_connections 2021-11-20T08:55:54.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:54.208+0000 7faaf064e700 1 -- 172.21.15.93:0/3125573024 wait complete. 2021-11-20T08:55:54.539 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":17,"active_gid":14162,"active_name":"smithi093.qdikkm","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":3953495879},{"type":"v1","addr":"172.21.15.93:6801","nonce":3953495879}]},"active_addr":"172.21.15.93:6801/3953495879","active_change":"2021-11-20T08:52:51.424929+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14198,"name":"smithi102.ievjft","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi093.front.sepia.ceph.com:8443/","prometheus":"http://smithi093.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.93:0","nonce":407566839}]},{"addrvec":[{"type":"v2","addr":"172.21.15.93:0","nonce":179929112}]}]}} 2021-11-20T08:55:54.546 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-11-20T08:55:54.546 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-11-20T08:55:54.547 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T08:55:54.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:54 smithi102 conmon[36472]: audit 2021-11-20T08:55:54.203809+0000 mon.smithi093 (mon.0) 433 : audit [DBG] from='client.? 172.21.15.93:0/3125573024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T08:55:54.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:54 smithi093 conmon[33522]: audit 2021-11-20T08:55:54.203809+0000 mon.smithi093 (mon.0) 433 : audit [DBG] from='client.? 172.21.15.93:0/3125573024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T08:55:54.990 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:55:56.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 -- 172.21.15.93:0/74177681 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f56d0 msgr2=0x7feda00f5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:56.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 --2- 172.21.15.93:0/74177681 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f56d0 0x7feda00f5af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fed90004660 tx=0x7fed90009b30).stop 2021-11-20T08:55:56.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 -- 172.21.15.93:0/74177681 shutdown_connections 2021-11-20T08:55:56.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 --2- 172.21.15.93:0/74177681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feda00f6970 0x7feda00f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:55:56.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 --2- 172.21.15.93:0/74177681 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f56d0 0x7feda00f5af0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:56.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 -- 172.21.15.93:0/74177681 >> 172.21.15.93:0/74177681 conn(0x7feda00f0e10 msgr2=0x7feda00f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:56.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 -- 172.21.15.93:0/74177681 shutdown_connections 2021-11-20T08:55:56.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.137+0000 7feda5016700 1 -- 172.21.15.93:0/74177681 wait complete. 2021-11-20T08:55:56.139 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:55 smithi093 conmon[33522]: audit 2021-11-20T08:55:54. 2021-11-20T08:55:56.140 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:55 smithi093 conmon[33522]: 918477+0000 mon.smithi093 (mon.0) 434 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:56.140 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:55 smithi093 conmon[33522]: audit 2021-11-20T08:55:54.921400+0000 mon.smithi093 (mon.0) 435 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:56.140 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:55 smithi093 conmon[33522]: cluster 2021-11-20T08:55:55.512451+0000 mgr.smithi093.qdikkm (mgr.14162) 148 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:56.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.138+0000 7feda5016700 1 Processor -- start 2021-11-20T08:55:56.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.138+0000 7feda5016700 1 -- start start 2021-11-20T08:55:56.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.138+0000 7feda5016700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feda00f56d0 0x7feda0105ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:56.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.138+0000 7feda5016700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f6970 0x7feda01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:56.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.138+0000 7feda5016700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda0106b20 con 0x7feda00f6970 2021-11-20T08:55:56.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.138+0000 7feda5016700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda01073d0 con 0x7feda00f56d0 2021-11-20T08:55:56.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f6970 0x7feda01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:56.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f6970 0x7feda01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42802/0 (socket says 172.21.15.93:42802) 2021-11-20T08:55:56.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9e59c700 1 -- 172.21.15.93:0/4285587464 learned_addr learned my addr 172.21.15.93:0/4285587464 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:55:56.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9ed9d700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feda00f56d0 0x7feda0105ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:56.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9e59c700 1 -- 172.21.15.93:0/4285587464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feda00f56d0 msgr2=0x7feda0105ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:56.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9e59c700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feda00f56d0 0x7feda0105ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:56.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9e59c700 1 -- 172.21.15.93:0/4285587464 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed90005040 con 0x7feda00f6970 2021-11-20T08:55:56.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9ed9d700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feda00f56d0 0x7feda0105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:55:56.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed9e59c700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f6970 0x7feda01064e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fed9400d770 tx=0x7fed9400db30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:56.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed877fe700 1 -- 172.21.15.93:0/4285587464 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed9400ea60 con 0x7feda00f6970 2021-11-20T08:55:56.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed877fe700 1 -- 172.21.15.93:0/4285587464 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed94012070 con 0x7feda00f6970 2021-11-20T08:55:56.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda0107650 con 0x7feda00f6970 2021-11-20T08:55:56.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7fed877fe700 1 -- 172.21.15.93:0/4285587464 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed94015330 con 0x7feda00f6970 2021-11-20T08:55:56.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.139+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda0107bb0 con 0x7feda00f6970 2021-11-20T08:55:56.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.140+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda0045ad0 con 0x7feda00f6970 2021-11-20T08:55:56.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.141+0000 7fed877fe700 1 -- 172.21.15.93:0/4285587464 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fed9401e070 con 0x7feda00f6970 2021-11-20T08:55:56.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.141+0000 7fed877fe700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed8805c2a0 0x7fed8805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:56.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.141+0000 7fed877fe700 1 -- 172.21.15.93:0/4285587464 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fed94077ec0 con 0x7feda00f6970 2021-11-20T08:55:56.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.141+0000 7fed9ed9d700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed8805c2a0 0x7fed8805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:56.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.141+0000 7fed9ed9d700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed8805c2a0 0x7fed8805e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fed90004e30 tx=0x7fed90009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:56.149 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.145+0000 7fed877fe700 1 -- 172.21.15.93:0/4285587464 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed9404aec0 con 0x7feda00f6970 2021-11-20T08:55:56.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:55 smithi102 conmon[36472]: audit 2021-11-20T08:55:54.918477+0000 mon.smithi093 (mon.0) 434 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:56.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:55 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:55:56.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:55 smithi102 conmon[36472]: 20T08:55:54.921400+0000 mon.smithi093 (mon.0) 435 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:55:56.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:55 smithi102 conmon[36472]: cluster 2021-11-20T08:55:55.512451+0000 mgr.smithi093.qdikkm (mgr.14162) 148 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.295+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feda00ffa00 con 0x7feda00f6970 2021-11-20T08:55:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.296+0000 7fed877fe700 1 -- 172.21.15.93:0/4285587464 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+10911 (secure 0 0 0) 0x7fed9404e4e0 con 0x7feda00f6970 2021-11-20T08:55:56.298 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:55:56.299 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":38,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","created":"2021-11-20T08:51:57.464444+0000","modified":"2021-11-20T08:55:50.156671+0000","last_up_change":"2021-11-20T08:55:49.151657+0000","last_in_change":"2021-11-20T08:55:32.747237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T08:55:09.134409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b62eaab5-818e-4739-98c3-d386425d4837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6803","nonce":1548414389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6805","nonce":1548414389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6809","nonce":1548414389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6807","nonce":1548414389}]},"public_addr":"172.21.15.93:6803/1548414389","cluster_addr":"172.21.15.93:6805/1548414389","heartbeat_back_addr":"172.21.15.93:6809/1548414389","heartbeat_front_addr":"172.21.15.93:6807/1548414389","state":["exists","up"]},{"osd":1,"uuid":"a46f3336-3bc4-4736-9095-c4d72e662323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6811","nonce":3785015356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6813","nonce":3785015356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6817","nonce":3785015356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6815","nonce":3785015356}]},"public_addr":"172.21.15.93:6811/3785015356","cluster_addr":"172.21.15.93:6813/3785015356","heartbeat_back_addr":"172.21.15.93:6817/3785015356","heartbeat_front_addr":"172.21.15.93:6815/3785015356","state":["exists","up"]},{"osd":2,"uuid":"72096543-af61-429d-a8d1-a77726a1f3eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6819","nonce":82486730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6821","nonce":82486730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6825","nonce":82486730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6823","nonce":82486730}]},"public_addr":"172.21.15.93:6819/82486730","cluster_addr":"172.21.15.93:6821/82486730","heartbeat_back_addr":"172.21.15.93:6825/82486730","heartbeat_front_addr":"172.21.15.93:6823/82486730","state":["exists","up"]},{"osd":3,"uuid":"37ba4275-7023-4f74-ba1f-7ad2f864476f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6801","nonce":1093924688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6803","nonce":1093924688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6807","nonce":1093924688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6805","nonce":1093924688}]},"public_addr":"172.21.15.102:6801/1093924688","cluster_addr":"172.21.15.102:6803/1093924688","heartbeat_back_addr":"172.21.15.102:6807/1093924688","heartbeat_front_addr":"172.21.15.102:6805/1093924688","state":["exists","up"]},{"osd":4,"uuid":"c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6809","nonce":2700333748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6811","nonce":2700333748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6815","nonce":2700333748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6813","nonce":2700333748}]},"public_addr":"172.21.15.102:6809/2700333748","cluster_addr":"172.21.15.102:6811/2700333748","heartbeat_back_addr":"172.21.15.102:6815/2700333748","heartbeat_front_addr":"172.21.15.102:6813/2700333748","state":["exists","up"]},{"osd":5,"uuid":"fab60808-4ace-4f26-9c42-f72340134a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6817","nonce":4083093586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6819","nonce":4083093586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6823","nonce":4083093586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6821","nonce":4083093586}]},"public_addr":"172.21.15.102:6817/4083093586","cluster_addr":"172.21.15.102:6819/4083093586","heartbeat_back_addr":"172.21.15.102:6823/4083093586","heartbeat_front_addr":"172.21.15.102:6821/4083093586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:21.010676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:41.314338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:57.961372+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:13.733626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:30.250474+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:47.331557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:0/3689853676":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/3631043468":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/485709266":"2021-11-21T08:52:37.986811+0000","172.21.15.93:6800/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1880974768":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1518372611":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/2364704567":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/1580490298":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6801/1580490298":"2021-11-21T08:52:15.847551+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T08:55:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed8805c2a0 msgr2=0x7fed8805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed8805c2a0 0x7fed8805e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fed90004e30 tx=0x7fed90009b30).stop 2021-11-20T08:55:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f6970 msgr2=0x7feda01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f6970 0x7feda01064e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fed9400d770 tx=0x7fed9400db30).stop 2021-11-20T08:55:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 shutdown_connections 2021-11-20T08:55:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fed8805c2a0 0x7fed8805e760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feda00f56d0 0x7feda0105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 --2- 172.21.15.93:0/4285587464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feda00f6970 0x7feda01064e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:56.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 >> 172.21.15.93:0/4285587464 conn(0x7feda00f0e10 msgr2=0x7feda00fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:56.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 shutdown_connections 2021-11-20T08:55:56.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:56.297+0000 7feda5016700 1 -- 172.21.15.93:0/4285587464 wait complete. 2021-11-20T08:55:56.687 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-11-20T08:55:56.688 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T08:55:57.162 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:55:57.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:56 smithi093 conmon[33522]: audit 2021-11-20T08:55:56.296434+0000 mon.smithi093 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.93:0/4285587464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:55:57.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:56 smithi093 conmon[33522]: audit 2021-11-20T08:55:56.618777+0000 mon.smithi093 (mon.0) 437 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:55:57.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:56 smithi093 conmon[33522]: audit 2021-11-20T08:55:56.618974+0000 mon.smithi093 (mon.0) 438 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:55:57.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:56 smithi102 conmon[36472]: audit 2021-11-20T08:55:56.296434+0000 mon.smithi093 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.93:0/4285587464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:55:57.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:56 smithi102 conmon[36472]: audit 2021-11-20T08:55:56.618777+0000 mon.smithi093 (mon.0) 437 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:55:57.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:56 smithi102 conmon[36472]: audit 2021-11-20T08:55:56.618974+0000 mon.smithi093 (mon.0) 438 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:55:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:57 smithi093 conmon[33522]: cluster 2021-11-20T08:55:57.512899+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:55:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:57 smithi093 conmon[33522]: 149 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:58.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.344+0000 7f8f8a18f700 1 -- 172.21.15.93:0/1202891147 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 msgr2=0x7f8f840f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:58.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.344+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/1202891147 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f840f8100 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8f80004660 tx=0x7f8f80009b30).stop 2021-11-20T08:55:58.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.345+0000 7f8f8a18f700 1 -- 172.21.15.93:0/1202891147 shutdown_connections 2021-11-20T08:55:58.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.345+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/1202891147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840f62f0 0x7f8f840f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:58.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.345+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/1202891147 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f840f8100 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:58.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.345+0000 7f8f8a18f700 1 -- 172.21.15.93:0/1202891147 >> 172.21.15.93:0/1202891147 conn(0x7f8f840f1d70 msgr2=0x7f8f840f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:58.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.345+0000 7f8f8a18f700 1 -- 172.21.15.93:0/1202891147 shutdown_connections 2021-11-20T08:55:58.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.345+0000 7f8f8a18f700 1 -- 172.21.15.93:0/1202891147 wait complete. 2021-11-20T08:55:58.348 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.345+0000 7f8f8a18f700 1 Processor -- start 2021-11-20T08:55:58.349 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8a18f700 1 -- start start 2021-11-20T08:55:58.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8a18f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840f62f0 0x7f8f8406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:58.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8a18f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f84069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:58.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8a18f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8406a380 con 0x7f8f840f7ce0 2021-11-20T08:55:58.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8a18f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8406a4c0 con 0x7f8f840f62f0 2021-11-20T08:55:58.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8898c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f84069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:58.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8898c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f84069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42812/0 (socket says 172.21.15.93:42812) 2021-11-20T08:55:58.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8898c700 1 -- 172.21.15.93:0/822120170 learned_addr learned my addr 172.21.15.93:0/822120170 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:55:58.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8918d700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840f62f0 0x7f8f8406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:58.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8898c700 1 -- 172.21.15.93:0/822120170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840f62f0 msgr2=0x7f8f8406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:58.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8898c700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840f62f0 0x7f8f8406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:58.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8898c700 1 -- 172.21.15.93:0/822120170 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80005040 con 0x7f8f840f7ce0 2021-11-20T08:55:58.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8918d700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840f62f0 0x7f8f8406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:55:58.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8898c700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f84069e20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8f7400c530 tx=0x7f8f74007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:58.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f79ffb700 1 -- 172.21.15.93:0/822120170 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f7400f650 con 0x7f8f840f7ce0 2021-11-20T08:55:58.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f79ffb700 1 -- 172.21.15.93:0/822120170 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f7400fc60 con 0x7f8f840f7ce0 2021-11-20T08:55:58.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8406a7a0 con 0x7f8f840f7ce0 2021-11-20T08:55:58.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.346+0000 7f8f79ffb700 1 -- 172.21.15.93:0/822120170 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f74012c80 con 0x7f8f840f7ce0 2021-11-20T08:55:58.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.347+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8410f000 con 0x7f8f840f7ce0 2021-11-20T08:55:58.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.348+0000 7f8f79ffb700 1 -- 172.21.15.93:0/822120170 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8f74010070 con 0x7f8f840f7ce0 2021-11-20T08:55:58.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.348+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f84045ad0 con 0x7f8f840f7ce0 2021-11-20T08:55:58.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.348+0000 7f8f79ffb700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8f7005c2b0 0x7f8f7005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:55:58.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.348+0000 7f8f79ffb700 1 -- 172.21.15.93:0/822120170 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8f740758f0 con 0x7f8f840f7ce0 2021-11-20T08:55:58.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.348+0000 7f8f8918d700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8f7005c2b0 0x7f8f7005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:55:58.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.348+0000 7f8f8918d700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8f7005c2b0 0x7f8f7005e770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8f8406b730 tx=0x7f8f80009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:55:58.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.352+0000 7f8f79ffb700 1 -- 172.21.15.93:0/822120170 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f740452b0 con 0x7f8f840f7ce0 2021-11-20T08:55:58.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:57 smithi102 conmon[36472]: cluster 2021-11-20T08:55:57.512899+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:55:58.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:57 smithi102 conmon[36472]: ) 149 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:55:58.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.503+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f84027a00 con 0x7f8f840f7ce0 2021-11-20T08:55:58.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.503+0000 7f8f79ffb700 1 -- 172.21.15.93:0/822120170 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+10911 (secure 0 0 0) 0x7f8f7404bf90 con 0x7f8f840f7ce0 2021-11-20T08:55:58.503 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:55:58.503 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":38,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","created":"2021-11-20T08:51:57.464444+0000","modified":"2021-11-20T08:55:50.156671+0000","last_up_change":"2021-11-20T08:55:49.151657+0000","last_in_change":"2021-11-20T08:55:32.747237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T08:55:09.134409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b62eaab5-818e-4739-98c3-d386425d4837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6803","nonce":1548414389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6805","nonce":1548414389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6809","nonce":1548414389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6807","nonce":1548414389}]},"public_addr":"172.21.15.93:6803/1548414389","cluster_addr":"172.21.15.93:6805/1548414389","heartbeat_back_addr":"172.21.15.93:6809/1548414389","heartbeat_front_addr":"172.21.15.93:6807/1548414389","state":["exists","up"]},{"osd":1,"uuid":"a46f3336-3bc4-4736-9095-c4d72e662323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6811","nonce":3785015356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6813","nonce":3785015356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6817","nonce":3785015356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6815","nonce":3785015356}]},"public_addr":"172.21.15.93:6811/3785015356","cluster_addr":"172.21.15.93:6813/3785015356","heartbeat_back_addr":"172.21.15.93:6817/3785015356","heartbeat_front_addr":"172.21.15.93:6815/3785015356","state":["exists","up"]},{"osd":2,"uuid":"72096543-af61-429d-a8d1-a77726a1f3eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6819","nonce":82486730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6821","nonce":82486730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6825","nonce":82486730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6823","nonce":82486730}]},"public_addr":"172.21.15.93:6819/82486730","cluster_addr":"172.21.15.93:6821/82486730","heartbeat_back_addr":"172.21.15.93:6825/82486730","heartbeat_front_addr":"172.21.15.93:6823/82486730","state":["exists","up"]},{"osd":3,"uuid":"37ba4275-7023-4f74-ba1f-7ad2f864476f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6801","nonce":1093924688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6803","nonce":1093924688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6807","nonce":1093924688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6805","nonce":1093924688}]},"public_addr":"172.21.15.102:6801/1093924688","cluster_addr":"172.21.15.102:6803/1093924688","heartbeat_back_addr":"172.21.15.102:6807/1093924688","heartbeat_front_addr":"172.21.15.102:6805/1093924688","state":["exists","up"]},{"osd":4,"uuid":"c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6809","nonce":2700333748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6811","nonce":2700333748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6815","nonce":2700333748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6813","nonce":2700333748}]},"public_addr":"172.21.15.102:6809/2700333748","cluster_addr":"172.21.15.102:6811/2700333748","heartbeat_back_addr":"172.21.15.102:6815/2700333748","heartbeat_front_addr":"172.21.15.102:6813/2700333748","state":["exists","up"]},{"osd":5,"uuid":"fab60808-4ace-4f26-9c42-f72340134a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6817","nonce":4083093586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6819","nonce":4083093586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6823","nonce":4083093586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6821","nonce":4083093586}]},"public_addr":"172.21.15.102:6817/4083093586","cluster_addr":"172.21.15.102:6819/4083093586","heartbeat_back_addr":"172.21.15.102:6823/4083093586","heartbeat_front_addr":"172.21.15.102:6821/4083093586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:21.010676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:41.314338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:57.961372+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:13.733626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:30.250474+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:47.331557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:0/3689853676":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/3631043468":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/485709266":"2021-11-21T08:52:37.986811+0000","172.21.15.93:6800/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1880974768":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1518372611":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/2364704567":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/1580490298":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6801/1580490298":"2021-11-21T08:52:15.847551+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T08:55:58.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8f7005c2b0 msgr2=0x7f8f7005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:58.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8f7005c2b0 0x7f8f7005e770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8f8406b730 tx=0x7f8f80009b30).stop 2021-11-20T08:55:58.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 msgr2=0x7f8f84069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:55:58.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f84069e20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8f7400c530 tx=0x7f8f74007e20).stop 2021-11-20T08:55:58.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 shutdown_connections 2021-11-20T08:55:58.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8f7005c2b0 0x7f8f7005e770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:58.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840f62f0 0x7f8f8406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:58.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 --2- 172.21.15.93:0/822120170 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f840f7ce0 0x7f8f84069e20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:55:58.507 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 >> 172.21.15.93:0/822120170 conn(0x7f8f840f1d70 msgr2=0x7f8f840fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:55:58.507 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 shutdown_connections 2021-11-20T08:55:58.507 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:55:58.505+0000 7f8f8a18f700 1 -- 172.21.15.93:0/822120170 wait complete. 2021-11-20T08:55:58.875 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-11-20T08:55:59.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:58 smithi093 conmon[33522]: audit 2021-11-20T08:55:58.504117+0000 2021-11-20T08:55:59.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:58 smithi093 conmon[33522]: mon.smithi093 (mon.0) 439 : audit [DBG] from='client.? 172.21.15.93:0/822120170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:55:59.308 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:55:59.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:58 smithi102 conmon[36472]: audit 2021-11-20T08:55:58.504117+0000 mon.smithi093 (mon.0) 2021-11-20T08:55:59.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:58 smithi102 conmon[36472]: 439 : audit [DBG] from='client.? 172.21.15.93:0/822120170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:55:59.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:55:59 smithi093 conmon[33522]: cluster 2021-11-20T08:55:59.513189+0000 mgr.smithi093.qdikkm (mgr.14162) 150 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:00.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/2497081647 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 msgr2=0x7f9e380f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:00.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/2497081647 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e380f4d00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9e34004660 tx=0x7f9e34009b30).stop 2021-11-20T08:56:00.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/2497081647 shutdown_connections 2021-11-20T08:56:00.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/2497081647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e380f5280 0x7f9e380f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/2497081647 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e380f4d00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/2497081647 >> 172.21.15.93:0/2497081647 conn(0x7f9e380f0400 msgr2=0x7f9e380f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:00.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/2497081647 shutdown_connections 2021-11-20T08:56:00.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.351+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/2497081647 wait complete. 2021-11-20T08:56:00.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3f2fa700 1 Processor -- start 2021-11-20T08:56:00.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3f2fa700 1 -- start start 2021-11-20T08:56:00.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3f2fa700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e38106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:00.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3f2fa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e380f5280 0x7f9e381073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:00.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3f2fa700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38107980 con 0x7f9e380f48e0 2021-11-20T08:56:00.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3f2fa700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38107ac0 con 0x7f9e380f5280 2021-11-20T08:56:00.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3e2f8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e38106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:00.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3e2f8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e38106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42820/0 (socket says 172.21.15.93:42820) 2021-11-20T08:56:00.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3e2f8700 1 -- 172.21.15.93:0/3593407961 learned_addr learned my addr 172.21.15.93:0/3593407961 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:00.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.352+0000 7f9e3daf7700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e380f5280 0x7f9e381073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:00.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e3e2f8700 1 -- 172.21.15.93:0/3593407961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e380f5280 msgr2=0x7f9e381073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:00.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e3e2f8700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e380f5280 0x7f9e381073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e3e2f8700 1 -- 172.21.15.93:0/3593407961 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34005040 con 0x7f9e380f48e0 2021-11-20T08:56:00.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e3e2f8700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e38106ee0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9e34004e30 tx=0x7f9e34009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:00.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e34018070 con 0x7f9e380f48e0 2021-11-20T08:56:00.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e34013410 con 0x7f9e380f48e0 2021-11-20T08:56:00.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e38108400 con 0x7f9e380f48e0 2021-11-20T08:56:00.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e3401de20 con 0x7f9e380f48e0 2021-11-20T08:56:00.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.353+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e38108960 con 0x7f9e380f48e0 2021-11-20T08:56:00.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.354+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e3805db60 con 0x7f9e380f48e0 2021-11-20T08:56:00.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.354+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9e3401d420 con 0x7f9e380f48e0 2021-11-20T08:56:00.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.354+0000 7f9e2effd700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9e2405c2b0 0x7f9e2405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:00.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9e34077800 con 0x7f9e380f48e0 2021-11-20T08:56:00.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e3daf7700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9e2405c2b0 0x7f9e2405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:00.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e2effd700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] conn(0x7f9e24062040 0x7f9e240644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:00.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 --> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e24064b20 con 0x7f9e24062040 2021-11-20T08:56:00.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e3407c220 con 0x7f9e380f48e0 2021-11-20T08:56:00.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e3eaf9700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] conn(0x7f9e24062040 0x7f9e240644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:00.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e3daf7700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9e2405c2b0 0x7f9e2405e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f9e380f56c0 tx=0x7f9e2800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:00.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.355+0000 7f9e3eaf9700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] conn(0x7f9e24062040 0x7f9e240644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:00.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.356+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== osd.0 v2:172.21.15.93:6802/1548414389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f9e24064b20 con 0x7f9e24062040 2021-11-20T08:56:00.368 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 --> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e38108dc0 con 0x7f9e24062040 2021-11-20T08:56:00.369 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e2effd700 1 -- 172.21.15.93:0/3593407961 <== osd.0 v2:172.21.15.93:6802/1548414389 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9e38108dc0 con 0x7f9e24062040 2021-11-20T08:56:00.371 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] conn(0x7f9e24062040 msgr2=0x7f9e240644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:00.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] conn(0x7f9e24062040 0x7f9e240644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9e2405c2b0 msgr2=0x7f9e2405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:00.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9e2405c2b0 0x7f9e2405e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f9e380f56c0 tx=0x7f9e2800b040).stop 2021-11-20T08:56:00.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 msgr2=0x7f9e38106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:00.373 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e38106ee0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9e34004e30 tx=0x7f9e34009b30).stop 2021-11-20T08:56:00.373 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 shutdown_connections 2021-11-20T08:56:00.373 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9e2405c2b0 0x7f9e2405e770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.373 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e380f48e0 0x7f9e38106ee0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.93:6802/1548414389,v1:172.21.15.93:6803/1548414389] conn(0x7f9e24062040 0x7f9e240644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 --2- 172.21.15.93:0/3593407961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e380f5280 0x7f9e381073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:00.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 >> 172.21.15.93:0/3593407961 conn(0x7f9e380f0400 msgr2=0x7f9e380fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:00.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 shutdown_connections 2021-11-20T08:56:00.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:00.370+0000 7f9e3f2fa700 1 -- 172.21.15.93:0/3593407961 wait complete. 2021-11-20T08:56:00.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:55:59 smithi102 conmon[36472]: cluster 2021-11-20T08:55:59.513189+0000 mgr.smithi093.qdikkm (mgr.14162) 150 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:00.820 INFO:teuthology.orchestra.run.smithi093.stdout:38654705687 2021-11-20T08:56:00.821 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-11-20T08:56:01.256 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:02.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 -- 172.21.15.93:0/1357633171 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406a3b0 msgr2=0x7fc2a406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:02.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/1357633171 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406a3b0 0x7fc2a406a830 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc298004660 tx=0x7fc298009b30).stop 2021-11-20T08:56:02.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 -- 172.21.15.93:0/1357633171 shutdown_connections 2021-11-20T08:56:02.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/1357633171 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406a3b0 0x7fc2a406a830 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/1357633171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a406b9c0 0x7fc2a4069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 -- 172.21.15.93:0/1357633171 >> 172.21.15.93:0/1357633171 conn(0x7fc2a40f1f20 msgr2=0x7fc2a40f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:02.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 -- 172.21.15.93:0/1357633171 shutdown_connections 2021-11-20T08:56:02.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.307+0000 7fc2ab74b700 1 -- 172.21.15.93:0/1357633171 wait complete. 2021-11-20T08:56:02.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2ab74b700 1 Processor -- start 2021-11-20T08:56:02.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2ab74b700 1 -- start start 2021-11-20T08:56:02.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2ab74b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a406a3b0 0x7fc2a410d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:02.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2ab74b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406b9c0 0x7fc2a410db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:02.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2ab74b700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a410e120 con 0x7fc2a406b9c0 2021-11-20T08:56:02.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2ab74b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a410e260 con 0x7fc2a406a3b0 2021-11-20T08:56:02.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2a8ce6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406b9c0 0x7fc2a410db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:02.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2a8ce6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406b9c0 0x7fc2a410db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42832/0 (socket says 172.21.15.93:42832) 2021-11-20T08:56:02.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2a8ce6700 1 -- 172.21.15.93:0/2185376281 learned_addr learned my addr 172.21.15.93:0/2185376281 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:02.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2a94e7700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a406a3b0 0x7fc2a410d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:02.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2a8ce6700 1 -- 172.21.15.93:0/2185376281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a406a3b0 msgr2=0x7fc2a410d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:02.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.308+0000 7fc2a8ce6700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a406a3b0 0x7fc2a410d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2a8ce6700 1 -- 172.21.15.93:0/2185376281 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc298005040 con 0x7fc2a406b9c0 2021-11-20T08:56:02.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2a94e7700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a406a3b0 0x7fc2a410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:56:02.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2a8ce6700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406b9c0 0x7fc2a410db70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fc298009f50 tx=0x7fc298009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:02.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc298018070 con 0x7fc2a406b9c0 2021-11-20T08:56:02.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a410eba0 con 0x7fc2a406b9c0 2021-11-20T08:56:02.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc298013410 con 0x7fc2a406b9c0 2021-11-20T08:56:02.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc29801c3c0 con 0x7fc2a406b9c0 2021-11-20T08:56:02.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.309+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a410f100 con 0x7fc2a406b9c0 2021-11-20T08:56:02.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.310+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc2a405dcf0 con 0x7fc2a406b9c0 2021-11-20T08:56:02.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.310+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc29801d420 con 0x7fc2a406b9c0 2021-11-20T08:56:02.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a1ffb700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc28c05c2a0 0x7fc28c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:02.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc298077860 con 0x7fc2a406b9c0 2021-11-20T08:56:02.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a94e7700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc28c05c2a0 0x7fc28c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:02.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a1ffb700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] conn(0x7fc28c062030 0x7fc28c064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:02.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 --> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc28c064b10 con 0x7fc28c062030 2021-11-20T08:56:02.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc29807c220 con 0x7fc2a406b9c0 2021-11-20T08:56:02.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a9ce8700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] conn(0x7fc28c062030 0x7fc28c064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:02.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a94e7700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc28c05c2a0 0x7fc28c05e760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fc294009740 tx=0x7fc294009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:02.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.311+0000 7fc2a9ce8700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] conn(0x7fc28c062030 0x7fc28c064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:02.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.312+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== osd.1 v2:172.21.15.93:6810/3785015356 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fc28c064b10 con 0x7fc28c062030 2021-11-20T08:56:02.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.326+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 --> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc2a410f5e0 con 0x7fc28c062030 2021-11-20T08:56:02.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.326+0000 7fc2a1ffb700 1 -- 172.21.15.93:0/2185376281 <== osd.1 v2:172.21.15.93:6810/3785015356 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc2a410f5e0 con 0x7fc28c062030 2021-11-20T08:56:02.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] conn(0x7fc28c062030 msgr2=0x7fc28c064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:02.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] conn(0x7fc28c062030 0x7fc28c064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc28c05c2a0 msgr2=0x7fc28c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:02.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc28c05c2a0 0x7fc28c05e760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fc294009740 tx=0x7fc294009be0).stop 2021-11-20T08:56:02.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406b9c0 msgr2=0x7fc2a410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:02.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406b9c0 0x7fc2a410db70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fc298009f50 tx=0x7fc298009b30).stop 2021-11-20T08:56:02.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 shutdown_connections 2021-11-20T08:56:02.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6810/3785015356,v1:172.21.15.93:6811/3785015356] conn(0x7fc28c062030 0x7fc28c064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc28c05c2a0 0x7fc28c05e760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a406a3b0 0x7fc2a410d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 --2- 172.21.15.93:0/2185376281 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a406b9c0 0x7fc2a410db70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:02.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 >> 172.21.15.93:0/2185376281 conn(0x7fc2a40f1f20 msgr2=0x7fc2a40f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:02.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 shutdown_connections 2021-11-20T08:56:02.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:02.327+0000 7fc2ab74b700 1 -- 172.21.15.93:0/2185376281 wait complete. 2021-11-20T08:56:02.682 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:02 smithi093 conmon[33522]: cluster 2021-11-20T08:56:01.513500+0000 2021-11-20T08:56:02.682 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:02 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 151 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:02.766 INFO:teuthology.orchestra.run.smithi093.stdout:60129542162 2021-11-20T08:56:02.767 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-11-20T08:56:02.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:02 smithi102 conmon[36472]: cluster 2021-11-20T08:56:01.513500+0000 mgr.smithi093.qdikkm (mgr.14162) 151 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:03.200 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:03.631 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:03 smithi093 conmon[33522]: cluster 2021-11-20T08:56:03.513872+0000 mgr.smithi093.qdikkm (mgr.14162) 152 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:03.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:03 smithi102 conmon[36472]: cluster 2021-11-20T08:56:03.513872+0000 mgr.smithi093.qdikkm (mgr.14162) 152 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:04.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 -- 172.21.15.93:0/913908581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 msgr2=0x7fa87c0f60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:04.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 --2- 172.21.15.93:0/913908581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c0f60a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa870004660 tx=0x7fa870009b30).stop 2021-11-20T08:56:04.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 -- 172.21.15.93:0/913908581 shutdown_connections 2021-11-20T08:56:04.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 --2- 172.21.15.93:0/913908581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c0f60a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:04.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 --2- 172.21.15.93:0/913908581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa87c0f5040 0x7fa87c0f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:04.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 -- 172.21.15.93:0/913908581 >> 172.21.15.93:0/913908581 conn(0x7fa87c0f0140 msgr2=0x7fa87c0f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:04.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 -- 172.21.15.93:0/913908581 shutdown_connections 2021-11-20T08:56:04.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 -- 172.21.15.93:0/913908581 wait complete. 2021-11-20T08:56:04.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.321+0000 7fa881d1d700 1 Processor -- start 2021-11-20T08:56:04.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa881d1d700 1 -- start start 2021-11-20T08:56:04.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa881d1d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa87c0f5040 0x7fa87c109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:04.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa881d1d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c10c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:04.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa881d1d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0f8160 con 0x7fa87c0f5a00 2021-11-20T08:56:04.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa881d1d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0f82a0 con 0x7fa87c0f5040 2021-11-20T08:56:04.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa87affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c10c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:04.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa87affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c10c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42842/0 (socket says 172.21.15.93:42842) 2021-11-20T08:56:04.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa87affd700 1 -- 172.21.15.93:0/2678464880 learned_addr learned my addr 172.21.15.93:0/2678464880 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:04.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa87b7fe700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa87c0f5040 0x7fa87c109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:04.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa87affd700 1 -- 172.21.15.93:0/2678464880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa87c0f5040 msgr2=0x7fa87c109dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:04.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa87affd700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa87c0f5040 0x7fa87c109dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:04.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.322+0000 7fa87affd700 1 -- 172.21.15.93:0/2678464880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870005040 con 0x7fa87c0f5a00 2021-11-20T08:56:04.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.323+0000 7fa87affd700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c10c2d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa870009f50 tx=0x7fa870009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.323+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa870018070 con 0x7fa87c0f5a00 2021-11-20T08:56:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.323+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa870013410 con 0x7fa87c0f5a00 2021-11-20T08:56:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.323+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c10c8d0 con 0x7fa87c0f5a00 2021-11-20T08:56:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.323+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa87001c620 con 0x7fa87c0f5a00 2021-11-20T08:56:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.323+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c10ce00 con 0x7fa87c0f5a00 2021-11-20T08:56:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.324+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa87c05dd50 con 0x7fa87c0f5a00 2021-11-20T08:56:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.324+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa87001d420 con 0x7fa87c0f5a00 2021-11-20T08:56:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.324+0000 7fa863fff700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa86405c2a0 0x7fa86405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.324+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa870077a20 con 0x7fa87c0f5a00 2021-11-20T08:56:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.324+0000 7fa87b7fe700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa86405c2a0 0x7fa86405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.325+0000 7fa863fff700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] conn(0x7fa864062030 0x7fa864064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.325+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 --> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa864064b10 con 0x7fa864062030 2021-11-20T08:56:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.325+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa87007c220 con 0x7fa87c0f5a00 2021-11-20T08:56:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.325+0000 7fa87bfff700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] conn(0x7fa864062030 0x7fa864064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:04.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.325+0000 7fa87b7fe700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa86405c2a0 0x7fa86405e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa87c0f7d50 tx=0x7fa86c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:04.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.325+0000 7fa87bfff700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] conn(0x7fa864062030 0x7fa864064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:04.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.326+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== osd.2 v2:172.21.15.93:6818/82486730 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fa864064b10 con 0x7fa864062030 2021-11-20T08:56:04.338 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 --> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa87c0fae00 con 0x7fa864062030 2021-11-20T08:56:04.339 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa863fff700 1 -- 172.21.15.93:0/2678464880 <== osd.2 v2:172.21.15.93:6818/82486730 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa87c0fae00 con 0x7fa864062030 2021-11-20T08:56:04.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] conn(0x7fa864062030 msgr2=0x7fa864064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:04.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] conn(0x7fa864062030 0x7fa864064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:04.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa86405c2a0 msgr2=0x7fa86405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:04.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa86405c2a0 0x7fa86405e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa87c0f7d50 tx=0x7fa86c008040).stop 2021-11-20T08:56:04.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 msgr2=0x7fa87c10c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:04.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c10c2d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa870009f50 tx=0x7fa870009b30).stop 2021-11-20T08:56:04.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 shutdown_connections 2021-11-20T08:56:04.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6818/82486730,v1:172.21.15.93:6819/82486730] conn(0x7fa864062030 0x7fa864064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:04.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa86405c2a0 0x7fa86405e760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:04.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa87c0f5040 0x7fa87c109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:04.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 --2- 172.21.15.93:0/2678464880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa87c0f5a00 0x7fa87c10c2d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:04.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 >> 172.21.15.93:0/2678464880 conn(0x7fa87c0f0140 msgr2=0x7fa87c0fa430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:04.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 shutdown_connections 2021-11-20T08:56:04.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:04.340+0000 7fa881d1d700 1 -- 172.21.15.93:0/2678464880 wait complete. 2021-11-20T08:56:04.700 INFO:teuthology.orchestra.run.smithi093.stdout:81604378640 2021-11-20T08:56:04.701 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-11-20T08:56:05.137 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:05.898 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:05 smithi093 conmon[33522]: cluster 2021-11-20T08:56:05.514356+0000 mgr.smithi093.qdikkm (mgr.14162) 153 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:05.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:05 smithi102 conmon[36472]: cluster 2021-11-20T08:56:05.514356+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:56:05.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:05 smithi102 conmon[36472]: 153 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:06.215 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 -- 172.21.15.93:0/64456266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f6980 msgr2=0x7f8cc40f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:06.216 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/64456266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f6980 0x7f8cc40f6de0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8cb8004660 tx=0x7f8cb8009b30).stop 2021-11-20T08:56:06.216 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 -- 172.21.15.93:0/64456266 shutdown_connections 2021-11-20T08:56:06.216 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/64456266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f6980 0x7f8cc40f6de0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:06.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/64456266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40f56e0 0x7f8cc40f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:06.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 -- 172.21.15.93:0/64456266 >> 172.21.15.93:0/64456266 conn(0x7f8cc40f0e20 msgr2=0x7f8cc40f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:06.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 -- 172.21.15.93:0/64456266 shutdown_connections 2021-11-20T08:56:06.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 -- 172.21.15.93:0/64456266 wait complete. 2021-11-20T08:56:06.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.150+0000 7f8cc9c85700 1 Processor -- start 2021-11-20T08:56:06.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc9c85700 1 -- start start 2021-11-20T08:56:06.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc9c85700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f56e0 0x7f8cc410c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:06.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc9c85700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40f6980 0x7f8cc410e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:06.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc9c85700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4060810 con 0x7f8cc40f56e0 2021-11-20T08:56:06.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc9c85700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4060950 con 0x7f8cc40f6980 2021-11-20T08:56:06.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f56e0 0x7f8cc410c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:06.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f56e0 0x7f8cc410c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42852/0 (socket says 172.21.15.93:42852) 2021-11-20T08:56:06.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc37fe700 1 -- 172.21.15.93:0/1751379386 learned_addr learned my addr 172.21.15.93:0/1751379386 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:06.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc37fe700 1 -- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40f6980 msgr2=0x7f8cc410e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:06.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc37fe700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40f6980 0x7f8cc410e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:06.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc37fe700 1 -- 172.21.15.93:0/1751379386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb8005040 con 0x7f8cc40f56e0 2021-11-20T08:56:06.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.151+0000 7f8cc37fe700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f56e0 0x7f8cc410c210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8cb40093f0 tx=0x7f8cb4006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:06.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.152+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cb4009ea0 con 0x7f8cc40f56e0 2021-11-20T08:56:06.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.152+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cb4013af0 con 0x7f8cc40f56e0 2021-11-20T08:56:06.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.152+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc410ec00 con 0x7f8cc40f56e0 2021-11-20T08:56:06.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.152+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cb400dad0 con 0x7f8cc40f56e0 2021-11-20T08:56:06.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.152+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc410f0e0 con 0x7f8cc40f56e0 2021-11-20T08:56:06.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8cc405dcf0 con 0x7f8cc40f56e0 2021-11-20T08:56:06.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8cb401a060 con 0x7f8cc40f56e0 2021-11-20T08:56:06.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cabfff700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8cac05c2a0 0x7f8cac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:06.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8cb4075c00 con 0x7f8cc40f56e0 2021-11-20T08:56:06.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cc2ffd700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8cac05c2a0 0x7f8cac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:06.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cabfff700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] conn(0x7f8cac062030 0x7f8cac064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:06.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 --> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cac064b10 con 0x7f8cac062030 2021-11-20T08:56:06.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.153+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb4078270 con 0x7f8cc40f56e0 2021-11-20T08:56:06.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.154+0000 7f8cc3fff700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] conn(0x7f8cac062030 0x7f8cac064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:06.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.154+0000 7f8cc2ffd700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8cac05c2a0 0x7f8cac05e760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8cb8009f50 tx=0x7f8cb8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:06.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.154+0000 7f8cc3fff700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] conn(0x7f8cac062030 0x7f8cac064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:06.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.155+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== osd.3 v2:172.21.15.102:6800/1093924688 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f8cac064b10 con 0x7f8cac062030 2021-11-20T08:56:06.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.169+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 --> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8cc40581d0 con 0x7f8cac062030 2021-11-20T08:56:06.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.169+0000 7f8cabfff700 1 -- 172.21.15.93:0/1751379386 <== osd.3 v2:172.21.15.102:6800/1093924688 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8cc40581d0 con 0x7f8cac062030 2021-11-20T08:56:06.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] conn(0x7f8cac062030 msgr2=0x7f8cac064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:06.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] conn(0x7f8cac062030 0x7f8cac064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:06.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8cac05c2a0 msgr2=0x7f8cac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:06.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8cac05c2a0 0x7f8cac05e760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8cb8009f50 tx=0x7f8cb8009b30).stop 2021-11-20T08:56:06.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f56e0 msgr2=0x7f8cc410c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:06.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f56e0 0x7f8cc410c210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8cb40093f0 tx=0x7f8cb4006ec0).stop 2021-11-20T08:56:06.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 shutdown_connections 2021-11-20T08:56:06.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:6800/1093924688,v1:172.21.15.102:6801/1093924688] conn(0x7f8cac062030 0x7f8cac064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:06.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8cac05c2a0 0x7f8cac05e760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:06.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8cc40f56e0 0x7f8cc410c210 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:06.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 --2- 172.21.15.93:0/1751379386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40f6980 0x7f8cc410e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:06.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 >> 172.21.15.93:0/1751379386 conn(0x7f8cc40f0e20 msgr2=0x7f8cc40fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:06.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 shutdown_connections 2021-11-20T08:56:06.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:06.170+0000 7f8cc9c85700 1 -- 172.21.15.93:0/1751379386 wait complete. 2021-11-20T08:56:06.736 INFO:teuthology.orchestra.run.smithi093.stdout:115964117005 2021-11-20T08:56:06.736 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-11-20T08:56:07.171 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:07.891 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:07 smithi093 conmon[33522]: cluster 2021-11-20T08:56:07.514681+0000 mgr.smithi093.qdikkm (mgr.14162) 154 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:07.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:07 smithi102 conmon[36472]: cluster 2021-11-20T08:56:07.514681+0000 mgr.smithi093.qdikkm (mgr.14162) 154 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:08.338 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 -- 172.21.15.93:0/845159777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 msgr2=0x7fab040f5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:08.338 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 --2- 172.21.15.93:0/845159777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab040f5280 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7faaf8004660 tx=0x7faaf8009b30).stop 2021-11-20T08:56:08.338 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 -- 172.21.15.93:0/845159777 shutdown_connections 2021-11-20T08:56:08.339 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 --2- 172.21.15.93:0/845159777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab040f6060 0x7fab040f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:08.339 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 --2- 172.21.15.93:0/845159777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab040f5280 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:08.339 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 -- 172.21.15.93:0/845159777 >> 172.21.15.93:0/845159777 conn(0x7fab040f05e0 msgr2=0x7fab040f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:08.339 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 -- 172.21.15.93:0/845159777 shutdown_connections 2021-11-20T08:56:08.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 -- 172.21.15.93:0/845159777 wait complete. 2021-11-20T08:56:08.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.334+0000 7fab0acdf700 1 Processor -- start 2021-11-20T08:56:08.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab0acdf700 1 -- start start 2021-11-20T08:56:08.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab0acdf700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab041070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:08.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab0acdf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab040f6060 0x7fab041075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:08.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab0acdf700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04107c00 con 0x7fab040f4e60 2021-11-20T08:56:08.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab0acdf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab041084b0 con 0x7fab040f6060 2021-11-20T08:56:08.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab08a7b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab041070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:08.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab08a7b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab041070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42864/0 (socket says 172.21.15.93:42864) 2021-11-20T08:56:08.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab08a7b700 1 -- 172.21.15.93:0/2255532602 learned_addr learned my addr 172.21.15.93:0/2255532602 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:08.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab03fff700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab040f6060 0x7fab041075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:08.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab08a7b700 1 -- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab040f6060 msgr2=0x7fab041075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:08.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab08a7b700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab040f6060 0x7fab041075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:08.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab08a7b700 1 -- 172.21.15.93:0/2255532602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf8005040 con 0x7fab040f4e60 2021-11-20T08:56:08.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab08a7b700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab041070d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7faaf8009990 tx=0x7faaf8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:08.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.335+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaf8007ba0 con 0x7fab040f4e60 2021-11-20T08:56:08.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.336+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faaf8007d00 con 0x7fab040f4e60 2021-11-20T08:56:08.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.336+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaf8011630 con 0x7fab040f4e60 2021-11-20T08:56:08.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.336+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab04108730 con 0x7fab040f4e60 2021-11-20T08:56:08.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.336+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab04108c30 con 0x7fab040f4e60 2021-11-20T08:56:08.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab0405dcf0 con 0x7fab040f4e60 2021-11-20T08:56:08.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faaf80118a0 con 0x7fab040f4e60 2021-11-20T08:56:08.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab017fa700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faaf405c2a0 0x7faaf405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:08.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7faaf8077be0 con 0x7fab040f4e60 2021-11-20T08:56:08.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab03fff700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faaf405c2a0 0x7faaf405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:08.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab017fa700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] conn(0x7faaf4062030 0x7faaf4064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:08.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 --> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faaf4064b10 con 0x7faaf4062030 2021-11-20T08:56:08.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.337+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaf807a250 con 0x7fab040f4e60 2021-11-20T08:56:08.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.338+0000 7fab03fff700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faaf405c2a0 0x7faaf405e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7faaf000b790 tx=0x7faaf0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:08.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.338+0000 7fab0927c700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] conn(0x7faaf4062030 0x7faaf4064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:08.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.338+0000 7fab0927c700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] conn(0x7faaf4062030 0x7faaf4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:08.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.339+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== osd.4 v2:172.21.15.102:6808/2700333748 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7faaf4064b10 con 0x7faaf4062030 2021-11-20T08:56:08.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.353+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 --> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab04109110 con 0x7faaf4062030 2021-11-20T08:56:08.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab017fa700 1 -- 172.21.15.93:0/2255532602 <== osd.4 v2:172.21.15.102:6808/2700333748 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fab04109110 con 0x7faaf4062030 2021-11-20T08:56:08.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] conn(0x7faaf4062030 msgr2=0x7faaf4064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:08.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] conn(0x7faaf4062030 0x7faaf4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:08.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faaf405c2a0 msgr2=0x7faaf405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:08.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faaf405c2a0 0x7faaf405e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7faaf000b790 tx=0x7faaf0008040).stop 2021-11-20T08:56:08.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 msgr2=0x7fab041070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:08.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab041070d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7faaf8009990 tx=0x7faaf8009b30).stop 2021-11-20T08:56:08.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 shutdown_connections 2021-11-20T08:56:08.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:6808/2700333748,v1:172.21.15.102:6809/2700333748] conn(0x7faaf4062030 0x7faaf4064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:08.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7faaf405c2a0 0x7faaf405e760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:08.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab040f4e60 0x7fab041070d0 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0x7faaf8009990 tx=0x7faaf8009b30).stop 2021-11-20T08:56:08.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 --2- 172.21.15.93:0/2255532602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab040f6060 0x7fab041075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:08.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.354+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 >> 172.21.15.93:0/2255532602 conn(0x7fab040f05e0 msgr2=0x7fab040f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:08.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.355+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 shutdown_connections 2021-11-20T08:56:08.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:08.355+0000 7fab0acdf700 1 -- 172.21.15.93:0/2255532602 wait complete. 2021-11-20T08:56:08.678 INFO:teuthology.orchestra.run.smithi093.stdout:137438953482 2021-11-20T08:56:08.678 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-11-20T08:56:09.113 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:09.911 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:09 smithi093 conmon[33522]: cluster 2021-11-20T08:56:09.515014+0000 2021-11-20T08:56:09.911 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:09 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 155 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:09.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:09 smithi102 conmon[36472]: cluster 2021-11-20T08:56:09.515014+0000 mgr.smithi093.qdikkm (mgr.14162) 155 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:10.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- 172.21.15.93:0/2169049604 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 msgr2=0x7f22200f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:10.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2169049604 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f22200f7ee0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2214004660 tx=0x7f2214009b30).stop 2021-11-20T08:56:10.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- 172.21.15.93:0/2169049604 shutdown_connections 2021-11-20T08:56:10.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2169049604 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f22200f7ee0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2169049604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22200f67e0 0x7f22200f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- 172.21.15.93:0/2169049604 >> 172.21.15.93:0/2169049604 conn(0x7f22200f1f20 msgr2=0x7f22200f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:10.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- 172.21.15.93:0/2169049604 shutdown_connections 2021-11-20T08:56:10.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- 172.21.15.93:0/2169049604 wait complete. 2021-11-20T08:56:10.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 Processor -- start 2021-11-20T08:56:10.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- start start 2021-11-20T08:56:10.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22200f67e0 0x7f222006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:10.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f2220069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:10.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222006a380 con 0x7f22200f7a80 2021-11-20T08:56:10.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.163+0000 7f2224bbc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222006a4c0 con 0x7f22200f67e0 2021-11-20T08:56:10.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221dd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f2220069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:10.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221dd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f2220069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42876/0 (socket says 172.21.15.93:42876) 2021-11-20T08:56:10.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221dd9b700 1 -- 172.21.15.93:0/2955406919 learned_addr learned my addr 172.21.15.93:0/2955406919 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:10.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221e59c700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22200f67e0 0x7f222006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:10.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221dd9b700 1 -- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22200f67e0 msgr2=0x7f222006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:10.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221dd9b700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22200f67e0 0x7f222006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221dd9b700 1 -- 172.21.15.93:0/2955406919 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2214005040 con 0x7f22200f7a80 2021-11-20T08:56:10.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f221dd9b700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f2220069e20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2214009f50 tx=0x7f2214009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:10.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2214018070 con 0x7f22200f7a80 2021-11-20T08:56:10.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222006a740 con 0x7f22200f7a80 2021-11-20T08:56:10.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2214013410 con 0x7f22200f7a80 2021-11-20T08:56:10.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f221401c3c0 con 0x7f22200f7a80 2021-11-20T08:56:10.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.164+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222006aca0 con 0x7f22200f7a80 2021-11-20T08:56:10.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.165+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f222005dcf0 con 0x7f22200f7a80 2021-11-20T08:56:10.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.166+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f221401d420 con 0x7f22200f7a80 2021-11-20T08:56:10.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.166+0000 7f220effd700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f220805c2f0 0x7f220805e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:10.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.166+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2214077fb0 con 0x7f22200f7a80 2021-11-20T08:56:10.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.166+0000 7f221e59c700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f220805c2f0 0x7f220805e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:10.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.166+0000 7f220effd700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] conn(0x7f2208062080 0x7f22080644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:10.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.166+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 --> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2208064b60 con 0x7f2208062080 2021-11-20T08:56:10.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.166+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22140782e0 con 0x7f22200f7a80 2021-11-20T08:56:10.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.167+0000 7f221e59c700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f220805c2f0 0x7f220805e7b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f222006b730 tx=0x7f2218009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:10.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.167+0000 7f221ed9d700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] conn(0x7f2208062080 0x7f22080644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:10.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.167+0000 7f221ed9d700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] conn(0x7f2208062080 0x7f22080644e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:10.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.168+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== osd.5 v2:172.21.15.102:6816/4083093586 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f2208064b60 con 0x7f2208062080 2021-11-20T08:56:10.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.182+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 --> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2220058150 con 0x7f2208062080 2021-11-20T08:56:10.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.182+0000 7f220effd700 1 -- 172.21.15.93:0/2955406919 <== osd.5 v2:172.21.15.102:6816/4083093586 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2220058150 con 0x7f2208062080 2021-11-20T08:56:10.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] conn(0x7f2208062080 msgr2=0x7f22080644e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:10.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] conn(0x7f2208062080 0x7f22080644e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f220805c2f0 msgr2=0x7f220805e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:10.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f220805c2f0 0x7f220805e7b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f222006b730 tx=0x7f2218009040).stop 2021-11-20T08:56:10.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 msgr2=0x7f2220069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:10.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f2220069e20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2214009f50 tx=0x7f2214009b30).stop 2021-11-20T08:56:10.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 shutdown_connections 2021-11-20T08:56:10.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:6816/4083093586,v1:172.21.15.102:6817/4083093586] conn(0x7f2208062080 0x7f22080644e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f220805c2f0 0x7f220805e7b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22200f67e0 0x7f222006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 --2- 172.21.15.93:0/2955406919 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22200f7a80 0x7f2220069e20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:10.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 >> 172.21.15.93:0/2955406919 conn(0x7f22200f1f20 msgr2=0x7f22200fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:10.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 shutdown_connections 2021-11-20T08:56:10.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:10.183+0000 7f2224bbc700 1 -- 172.21.15.93:0/2955406919 wait complete. 2021-11-20T08:56:10.512 INFO:teuthology.orchestra.run.smithi093.stdout:158913789960 2021-11-20T08:56:10.514 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-11-20T08:56:10.947 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:11.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 -- 172.21.15.93:0/3287085267 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 msgr2=0x7f287c0f7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:11.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 --2- 172.21.15.93:0/3287085267 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c0f7d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f286c0045a0 tx=0x7f286c009b30).stop 2021-11-20T08:56:11.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 -- 172.21.15.93:0/3287085267 shutdown_connections 2021-11-20T08:56:11.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 --2- 172.21.15.93:0/3287085267 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c0f7d20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:11.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 --2- 172.21.15.93:0/3287085267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f287c0f6620 0x7f287c0f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:11.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 -- 172.21.15.93:0/3287085267 >> 172.21.15.93:0/3287085267 conn(0x7f287c0f1da0 msgr2=0x7f287c0f41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:11.992 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 -- 172.21.15.93:0/3287085267 shutdown_connections 2021-11-20T08:56:11.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.985+0000 7f288270c700 1 -- 172.21.15.93:0/3287085267 wait complete. 2021-11-20T08:56:11.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f288270c700 1 Processor -- start 2021-11-20T08:56:11.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f288270c700 1 -- start start 2021-11-20T08:56:11.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f288270c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f287c0f6620 0x7f287c106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:11.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f288270c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:11.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f288270c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c107910 con 0x7f287c0f78c0 2021-11-20T08:56:11.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f288270c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c107a50 con 0x7f287c0f6620 2021-11-20T08:56:11.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2880f09700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:11.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2880f09700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c107360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42888/0 (socket says 172.21.15.93:42888) 2021-11-20T08:56:11.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2880f09700 1 -- 172.21.15.93:0/3646488666 learned_addr learned my addr 172.21.15.93:0/3646488666 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:11.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f288170a700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f287c0f6620 0x7f287c106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:11.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2880f09700 1 -- 172.21.15.93:0/3646488666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f287c0f6620 msgr2=0x7f287c106e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:11.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2880f09700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f287c0f6620 0x7f287c106e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:11.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2880f09700 1 -- 172.21.15.93:0/3646488666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c005040 con 0x7f287c0f78c0 2021-11-20T08:56:11.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2880f09700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c107360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f286c002040 tx=0x7f286c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:11.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.986+0000 7f2871ffb700 1 -- 172.21.15.93:0/3646488666 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286c00e070 con 0x7f287c0f78c0 2021-11-20T08:56:11.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.987+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287c108390 con 0x7f287c0f78c0 2021-11-20T08:56:11.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.987+0000 7f2871ffb700 1 -- 172.21.15.93:0/3646488666 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f286c007ba0 con 0x7f287c0f78c0 2021-11-20T08:56:11.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.987+0000 7f2871ffb700 1 -- 172.21.15.93:0/3646488666 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286c019070 con 0x7f287c0f78c0 2021-11-20T08:56:11.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.987+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287c108870 con 0x7f287c0f78c0 2021-11-20T08:56:11.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.988+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287c045ad0 con 0x7f287c0f78c0 2021-11-20T08:56:12.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.988+0000 7f2871ffb700 1 -- 172.21.15.93:0/3646488666 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f286c014420 con 0x7f287c0f78c0 2021-11-20T08:56:12.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.988+0000 7f2871ffb700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f286805c2b0 0x7f286805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:12.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.988+0000 7f2871ffb700 1 -- 172.21.15.93:0/3646488666 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f286c076a40 con 0x7f287c0f78c0 2021-11-20T08:56:12.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.988+0000 7f288170a700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f286805c2b0 0x7f286805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:12.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.989+0000 7f288170a700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f286805c2b0 0x7f286805e770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2878009880 tx=0x7f287800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:12.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:11.992+0000 7f2871ffb700 1 -- 172.21.15.93:0/3646488666 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f286c049ac0 con 0x7f287c0f78c0 2021-11-20T08:56:12.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.143+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f287c027a00 con 0x7f287c0f78c0 2021-11-20T08:56:12.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.143+0000 7f2871ffb700 1 -- 172.21.15.93:0/3646488666 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f286c04d0e0 con 0x7f287c0f78c0 2021-11-20T08:56:12.142 INFO:teuthology.orchestra.run.smithi093.stdout:38654705689 2021-11-20T08:56:12.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f286805c2b0 msgr2=0x7f286805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:12.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f286805c2b0 0x7f286805e770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2878009880 tx=0x7f287800b040).stop 2021-11-20T08:56:12.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 msgr2=0x7f287c107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:12.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c107360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f286c002040 tx=0x7f286c00b040).stop 2021-11-20T08:56:12.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 shutdown_connections 2021-11-20T08:56:12.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f286805c2b0 0x7f286805e770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:12.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f287c0f6620 0x7f287c106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:12.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 --2- 172.21.15.93:0/3646488666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f287c0f78c0 0x7f287c107360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:12.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 >> 172.21.15.93:0/3646488666 conn(0x7f287c0f1da0 msgr2=0x7f287c0fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:12.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 shutdown_connections 2021-11-20T08:56:12.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:12.145+0000 7f288270c700 1 -- 172.21.15.93:0/3646488666 wait complete. 2021-11-20T08:56:12.453 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705687 got 38654705689 for osd.0 2021-11-20T08:56:12.454 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-11-20T08:56:12.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:12 smithi093 conmon[33522]: cluster 2021-11-20T08:56:11.515500+0000 mgr.smithi093.qdikkm (mgr.14162) 156 2021-11-20T08:56:12.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:12 smithi093 conmon[33522]: : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:12.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:12 smithi093 conmon[33522]: audit 2021-11-20T08:56:12.144064+0000 mon.smithi093 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.93:0/3646488666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T08:56:12.893 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:12.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:12 smithi102 conmon[36472]: cluster 2021-11-20T08:56:11.515500+0000 mgr.smithi093.qdikkm (mgr.14162) 156 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:12.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:12 smithi102 conmon[36472]: audit 2021-11-20T08:56:12.144064+0000 mon.smithi093 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.93:0/3646488666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T08:56:13.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:13 smithi093 conmon[33522]: cluster 2021-11-20T08:56:13.515848+0000 mgr.smithi093.qdikkm (mgr.14162) 157 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:13.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:13 smithi102 conmon[36472]: cluster 2021-11-20T08:56:13.515848+0000 mgr.smithi093.qdikkm (mgr.14162) 157 : cluster 2021-11-20T08:56:13.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:13 smithi102 conmon[36472]: [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:14.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2967069418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f6980 msgr2=0x7f7ed00f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:14.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2967069418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f6980 0x7f7ed00f6de0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f7ec0004660 tx=0x7f7ec0009b30).stop 2021-11-20T08:56:14.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2967069418 shutdown_connections 2021-11-20T08:56:14.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2967069418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f6980 0x7f7ed00f6de0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:14.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2967069418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ed00f56e0 0x7f7ed00f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:14.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2967069418 >> 172.21.15.93:0/2967069418 conn(0x7f7ed00f0e20 msgr2=0x7f7ed00f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:14.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2967069418 shutdown_connections 2021-11-20T08:56:14.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.036+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2967069418 wait complete. 2021-11-20T08:56:14.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed83f1700 1 Processor -- start 2021-11-20T08:56:14.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed83f1700 1 -- start start 2021-11-20T08:56:14.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed83f1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f56e0 0x7f7ed0109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:14.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed83f1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ed00f6980 0x7f7ed010c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:14.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed83f1700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0060940 con 0x7f7ed00f56e0 2021-11-20T08:56:14.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed83f1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0060a80 con 0x7f7ed00f6980 2021-11-20T08:56:14.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed618d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f56e0 0x7f7ed0109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:14.041 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed618d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f56e0 0x7f7ed0109da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42896/0 (socket says 172.21.15.93:42896) 2021-11-20T08:56:14.041 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed618d700 1 -- 172.21.15.93:0/2783361622 learned_addr learned my addr 172.21.15.93:0/2783361622 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:14.041 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed618d700 1 -- 172.21.15.93:0/2783361622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ed00f6980 msgr2=0x7f7ed010c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:14.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed618d700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ed00f6980 0x7f7ed010c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:14.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed618d700 1 -- 172.21.15.93:0/2783361622 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec0005040 con 0x7f7ed00f56e0 2021-11-20T08:56:14.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ed618d700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f56e0 0x7f7ed0109da0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7ecc00c790 tx=0x7f7ecc007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:14.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.037+0000 7f7ec6ffd700 1 -- 172.21.15.93:0/2783361622 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ecc00e680 con 0x7f7ed00f56e0 2021-11-20T08:56:14.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.038+0000 7f7ec6ffd700 1 -- 172.21.15.93:0/2783361622 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ecc00ec90 con 0x7f7ed00f56e0 2021-11-20T08:56:14.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.038+0000 7f7ec6ffd700 1 -- 172.21.15.93:0/2783361622 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ecc012cb0 con 0x7f7ed00f56e0 2021-11-20T08:56:14.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.038+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed010c850 con 0x7f7ed00f56e0 2021-11-20T08:56:14.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.038+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed010ce10 con 0x7f7ed00f56e0 2021-11-20T08:56:14.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.038+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed0045ad0 con 0x7f7ed00f56e0 2021-11-20T08:56:14.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.039+0000 7f7ec6ffd700 1 -- 172.21.15.93:0/2783361622 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7ecc010070 con 0x7f7ed00f56e0 2021-11-20T08:56:14.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.039+0000 7f7ec6ffd700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ebc05c2a0 0x7f7ebc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:14.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.039+0000 7f7ec6ffd700 1 -- 172.21.15.93:0/2783361622 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7ecc075bb0 con 0x7f7ed00f56e0 2021-11-20T08:56:14.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.039+0000 7f7ed598c700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ebc05c2a0 0x7f7ebc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:14.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.040+0000 7f7ed598c700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ebc05c2a0 0x7f7ebc05e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f7ec0009f50 tx=0x7f7ec0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:14.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.043+0000 7f7ec6ffd700 1 -- 172.21.15.93:0/2783361622 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ecc07c080 con 0x7f7ed00f56e0 2021-11-20T08:56:14.192 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.193+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7ed0027a00 con 0x7f7ed00f56e0 2021-11-20T08:56:14.193 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.193+0000 7f7ec6ffd700 1 -- 172.21.15.93:0/2783361622 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7ecc048bd0 con 0x7f7ed00f56e0 2021-11-20T08:56:14.193 INFO:teuthology.orchestra.run.smithi093.stdout:60129542164 2021-11-20T08:56:14.194 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ebc05c2a0 msgr2=0x7f7ebc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:14.194 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ebc05c2a0 0x7f7ebc05e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f7ec0009f50 tx=0x7f7ec0009b30).stop 2021-11-20T08:56:14.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f56e0 msgr2=0x7f7ed0109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:14.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f56e0 0x7f7ed0109da0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f7ecc00c790 tx=0x7f7ecc007d90).stop 2021-11-20T08:56:14.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 shutdown_connections 2021-11-20T08:56:14.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ebc05c2a0 0x7f7ebc05e760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:14.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ed00f56e0 0x7f7ed0109da0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:14.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 --2- 172.21.15.93:0/2783361622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ed00f6980 0x7f7ed010c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:14.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 >> 172.21.15.93:0/2783361622 conn(0x7f7ed00f0e20 msgr2=0x7f7ed00fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:14.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 shutdown_connections 2021-11-20T08:56:14.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:14.195+0000 7f7ed83f1700 1 -- 172.21.15.93:0/2783361622 wait complete. 2021-11-20T08:56:14.602 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542162 got 60129542164 for osd.1 2021-11-20T08:56:14.603 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-11-20T08:56:14.627 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:14 smithi093 conmon[33522]: audit 2021-11-20T08:56:14.194441+0000 mon.smithi093 2021-11-20T08:56:14.628 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:14 smithi093 conmon[33522]: (mon.0) 441 : audit [DBG] from='client.? 172.21.15.93:0/2783361622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T08:56:14.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:14 smithi102 conmon[36472]: audit 2021-11-20T08:56:14.194441+0000 mon.smithi093 (mon.0) 441 : audit [DBG] from='client.? 172.21.15.93:0/2783361622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T08:56:15.037 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:15.828 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:15 smithi093 conmon[33522]: cluster 2021-11-20T08:56:15.516180+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:56:15.828 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:15 smithi093 conmon[33522]: 14162) 158 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:15.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:15 smithi102 conmon[36472]: cluster 2021-11-20T08:56:15.516180+0000 mgr.smithi093.qdikkm (mgr.14162) 158 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:16.102 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 -- 172.21.15.93:0/834743059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f6980 msgr2=0x7f2f0c0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:16.103 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 --2- 172.21.15.93:0/834743059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f6980 0x7f2f0c0f6de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2efc004660 tx=0x7f2efc009b30).stop 2021-11-20T08:56:16.103 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 -- 172.21.15.93:0/834743059 shutdown_connections 2021-11-20T08:56:16.103 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 --2- 172.21.15.93:0/834743059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f6980 0x7f2f0c0f6de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:16.103 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 --2- 172.21.15.93:0/834743059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f0c0f56e0 0x7f2f0c0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:16.104 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 -- 172.21.15.93:0/834743059 >> 172.21.15.93:0/834743059 conn(0x7f2f0c0f0e20 msgr2=0x7f2f0c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:16.104 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 -- 172.21.15.93:0/834743059 shutdown_connections 2021-11-20T08:56:16.104 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.096+0000 7f2f13479700 1 -- 172.21.15.93:0/834743059 wait complete. 2021-11-20T08:56:16.105 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f13479700 1 Processor -- start 2021-11-20T08:56:16.105 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f13479700 1 -- start start 2021-11-20T08:56:16.106 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f13479700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f56e0 0x7f2f0c109d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:16.106 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f13479700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f0c0f6980 0x7f2f0c10c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:16.106 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f13479700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c060980 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.106 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f13479700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c060ac0 con 0x7f2f0c0f6980 2021-11-20T08:56:16.107 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f11215700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f56e0 0x7f2f0c109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:16.107 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f11215700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f56e0 0x7f2f0c109d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42904/0 (socket says 172.21.15.93:42904) 2021-11-20T08:56:16.107 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f11215700 1 -- 172.21.15.93:0/3675810829 learned_addr learned my addr 172.21.15.93:0/3675810829 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:16.107 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f10a14700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f0c0f6980 0x7f2f0c10c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:16.108 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f11215700 1 -- 172.21.15.93:0/3675810829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f0c0f6980 msgr2=0x7f2f0c10c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:16.108 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f11215700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f0c0f6980 0x7f2f0c10c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:16.108 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f11215700 1 -- 172.21.15.93:0/3675810829 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc005040 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.108 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.097+0000 7f2f11215700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f56e0 0x7f2f0c109d60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f2f0800c530 tx=0x7f2f08007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:16.109 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.098+0000 7f2f01ffb700 1 -- 172.21.15.93:0/3675810829 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f0800f650 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.109 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.098+0000 7f2f01ffb700 1 -- 172.21.15.93:0/3675810829 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f0800fc60 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.109 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.098+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c10c870 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.109 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.098+0000 7f2f01ffb700 1 -- 172.21.15.93:0/3675810829 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f08012d20 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.110 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.098+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c10ce00 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.110 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.099+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c059100 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.110 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.099+0000 7f2f01ffb700 1 -- 172.21.15.93:0/3675810829 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2f08019030 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.110 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.099+0000 7f2f01ffb700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2ef805c2a0 0x7f2ef805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:16.111 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.099+0000 7f2f01ffb700 1 -- 172.21.15.93:0/3675810829 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2f08075810 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.111 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.099+0000 7f2f10a14700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2ef805c2a0 0x7f2ef805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:16.111 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.100+0000 7f2f10a14700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2ef805c2a0 0x7f2ef805e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2efc009d60 tx=0x7f2efc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:16.111 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.103+0000 7f2f01ffb700 1 -- 172.21.15.93:0/3675810829 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f080462b0 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.253 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.254+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2f0c045ad0 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.253 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.254+0000 7f2f01ffb700 1 -- 172.21.15.93:0/3675810829 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f08017070 con 0x7f2f0c0f56e0 2021-11-20T08:56:16.253 INFO:teuthology.orchestra.run.smithi093.stdout:81604378643 2021-11-20T08:56:16.254 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.255+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2ef805c2a0 msgr2=0x7f2ef805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:16.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.255+0000 7f2f13479700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2ef805c2a0 0x7f2ef805e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2efc009d60 tx=0x7f2efc009b30).stop 2021-11-20T08:56:16.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.255+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f56e0 msgr2=0x7f2f0c109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:16.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.255+0000 7f2f13479700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f56e0 0x7f2f0c109d60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f2f0800c530 tx=0x7f2f08007e10).stop 2021-11-20T08:56:16.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.256+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 shutdown_connections 2021-11-20T08:56:16.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.256+0000 7f2f13479700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2ef805c2a0 0x7f2ef805e760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:16.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.256+0000 7f2f13479700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f0c0f56e0 0x7f2f0c109d60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:16.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.256+0000 7f2f13479700 1 --2- 172.21.15.93:0/3675810829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f0c0f6980 0x7f2f0c10c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:16.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.256+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 >> 172.21.15.93:0/3675810829 conn(0x7f2f0c0f0e20 msgr2=0x7f2f0c0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:16.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.256+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 shutdown_connections 2021-11-20T08:56:16.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:16.256+0000 7f2f13479700 1 -- 172.21.15.93:0/3675810829 wait complete. 2021-11-20T08:56:16.747 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378640 got 81604378643 for osd.2 2021-11-20T08:56:16.748 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-11-20T08:56:16.811 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:16 smithi093 conmon[33522]: audit 2021-11-20T08:56:16.255099+0000 2021-11-20T08:56:16.812 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:16 smithi093 conmon[33522]: mon.smithi093 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.93:0/3675810829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T08:56:16.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:16 smithi102 conmon[36472]: audit 2021-11-20T08:56:16.255099+0000 mon.smithi093 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.93:0/3675810829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T08:56:17.221 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:17.646 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:17 smithi093 conmon[33522]: cluster 2021-11-20T08:56:17.516673+0000 mgr.smithi093.qdikkm (mgr.14162) 159 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:17.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:17 smithi102 conmon[36472]: cluster 2021-11-20T08:56:17.516673+0000 mgr.smithi093.qdikkm (mgr.14162) 159 : cluster [DBG] 2021-11-20T08:56:17.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:17 smithi102 conmon[36472]: pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:18.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 -- 172.21.15.93:0/1372446552 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 msgr2=0x7f54580f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:18.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 --2- 172.21.15.93:0/1372446552 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54580f6c00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5454004660 tx=0x7f5454009b30).stop 2021-11-20T08:56:18.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 -- 172.21.15.93:0/1372446552 shutdown_connections 2021-11-20T08:56:18.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 --2- 172.21.15.93:0/1372446552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54580f7a80 0x7f54580f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:18.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 --2- 172.21.15.93:0/1372446552 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54580f6c00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:18.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 -- 172.21.15.93:0/1372446552 >> 172.21.15.93:0/1372446552 conn(0x7f54580f1f20 msgr2=0x7f54580f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:18.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 -- 172.21.15.93:0/1372446552 shutdown_connections 2021-11-20T08:56:18.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.353+0000 7f545fdfe700 1 -- 172.21.15.93:0/1372446552 wait complete. 2021-11-20T08:56:18.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545fdfe700 1 Processor -- start 2021-11-20T08:56:18.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545fdfe700 1 -- start start 2021-11-20T08:56:18.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545fdfe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54581070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:18.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545fdfe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54580f7a80 0x7f54581075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:18.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545fdfe700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5458107c10 con 0x7f54580f67e0 2021-11-20T08:56:18.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545fdfe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54581084c0 con 0x7f54580f7a80 2021-11-20T08:56:18.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545db9a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54581070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:18.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545db9a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54581070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42912/0 (socket says 172.21.15.93:42912) 2021-11-20T08:56:18.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545db9a700 1 -- 172.21.15.93:0/3882143386 learned_addr learned my addr 172.21.15.93:0/3882143386 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:18.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545d399700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54580f7a80 0x7f54581075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:18.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545db9a700 1 -- 172.21.15.93:0/3882143386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54580f7a80 msgr2=0x7f54581075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:18.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545db9a700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54580f7a80 0x7f54581075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:18.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545db9a700 1 -- 172.21.15.93:0/3882143386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454005040 con 0x7f54580f67e0 2021-11-20T08:56:18.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.354+0000 7f545d399700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54580f7a80 0x7f54581075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:56:18.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.355+0000 7f545db9a700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54581070e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5454009f50 tx=0x7f5454009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:18.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.355+0000 7f544e7fc700 1 -- 172.21.15.93:0/3882143386 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5454018070 con 0x7f54580f67e0 2021-11-20T08:56:18.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.355+0000 7f544e7fc700 1 -- 172.21.15.93:0/3882143386 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5454013410 con 0x7f54580f67e0 2021-11-20T08:56:18.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.355+0000 7f544e7fc700 1 -- 172.21.15.93:0/3882143386 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f545401c620 con 0x7f54580f67e0 2021-11-20T08:56:18.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.355+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5458108740 con 0x7f54580f67e0 2021-11-20T08:56:18.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.355+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5458108ca0 con 0x7f54580f67e0 2021-11-20T08:56:18.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.356+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5458059100 con 0x7f54580f67e0 2021-11-20T08:56:18.363 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.356+0000 7f544e7fc700 1 -- 172.21.15.93:0/3882143386 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f545401d420 con 0x7f54580f67e0 2021-11-20T08:56:18.363 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.356+0000 7f544e7fc700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f544405c2a0 0x7f544405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:18.363 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.356+0000 7f544e7fc700 1 -- 172.21.15.93:0/3882143386 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5454077a40 con 0x7f54580f67e0 2021-11-20T08:56:18.363 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.357+0000 7f545d399700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f544405c2a0 0x7f544405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:18.364 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.357+0000 7f545d399700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f544405c2a0 0x7f544405e760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f544800b830 tx=0x7f5448008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:18.364 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.360+0000 7f544e7fc700 1 -- 172.21.15.93:0/3882143386 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f545404aa40 con 0x7f54580f67e0 2021-11-20T08:56:18.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.512+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f54580ff0e0 con 0x7f54580f67e0 2021-11-20T08:56:18.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.513+0000 7f544e7fc700 1 -- 172.21.15.93:0/3882143386 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f545404e060 con 0x7f54580f67e0 2021-11-20T08:56:18.512 INFO:teuthology.orchestra.run.smithi093.stdout:115964117008 2021-11-20T08:56:18.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f544405c2a0 msgr2=0x7f544405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:18.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f544405c2a0 0x7f544405e760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f544800b830 tx=0x7f5448008040).stop 2021-11-20T08:56:18.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 msgr2=0x7f54581070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:18.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54581070e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5454009f50 tx=0x7f5454009b30).stop 2021-11-20T08:56:18.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 shutdown_connections 2021-11-20T08:56:18.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f544405c2a0 0x7f544405e760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:18.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54580f67e0 0x7f54581070e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:18.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 --2- 172.21.15.93:0/3882143386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54580f7a80 0x7f54581075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:18.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 >> 172.21.15.93:0/3882143386 conn(0x7f54580f1f20 msgr2=0x7f54580fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:18.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 shutdown_connections 2021-11-20T08:56:18.516 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:18.514+0000 7f545fdfe700 1 -- 172.21.15.93:0/3882143386 wait complete. 2021-11-20T08:56:18.660 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:18 smithi093 conmon[33522]: audit 2021-11-20T08:56:18.513823+0000 2021-11-20T08:56:18.660 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:18 smithi093 conmon[33522]: mon.smithi093 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.93:0/3882143386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T08:56:18.819 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117005 got 115964117008 for osd.3 2021-11-20T08:56:18.820 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-11-20T08:56:18.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:18 smithi102 conmon[36472]: audit 2021-11-20T08:56:18.513823+0000 mon.smithi093 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.93:0/3882143386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T08:56:19.293 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:19.686 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:19 smithi093 conmon[33522]: cluster 2021-11-20T08:56:19.516993 2021-11-20T08:56:19.698 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:19 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 160 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:19.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:19 smithi102 conmon[36472]: cluster 2021-11-20T08:56:19.516993+0000 mgr.smithi093.qdikkm (mgr.14162) 160 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:20.445 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 -- 172.21.15.93:0/236932360 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 msgr2=0x7f4c440f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:20.445 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 --2- 172.21.15.93:0/236932360 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c440f6be0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f4c34004660 tx=0x7f4c34009b30).stop 2021-11-20T08:56:20.445 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 -- 172.21.15.93:0/236932360 shutdown_connections 2021-11-20T08:56:20.446 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 --2- 172.21.15.93:0/236932360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c440f7a60 0x7f4c440f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:20.446 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 --2- 172.21.15.93:0/236932360 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c440f6be0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:20.446 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 -- 172.21.15.93:0/236932360 >> 172.21.15.93:0/236932360 conn(0x7f4c440f1f20 msgr2=0x7f4c440f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:20.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 -- 172.21.15.93:0/236932360 shutdown_connections 2021-11-20T08:56:20.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 -- 172.21.15.93:0/236932360 wait complete. 2021-11-20T08:56:20.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.443+0000 7f4c49c33700 1 Processor -- start 2021-11-20T08:56:20.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c49c33700 1 -- start start 2021-11-20T08:56:20.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c49c33700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c441070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:20.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c49c33700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c440f7a60 0x7f4c44107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:20.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c49c33700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c44107b40 con 0x7f4c440f67c0 2021-11-20T08:56:20.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c49c33700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c44107c80 con 0x7f4c440f7a60 2021-11-20T08:56:20.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c437fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c441070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:20.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c437fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c441070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42922/0 (socket says 172.21.15.93:42922) 2021-11-20T08:56:20.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c437fe700 1 -- 172.21.15.93:0/3613350992 learned_addr learned my addr 172.21.15.93:0/3613350992 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:20.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c437fe700 1 -- 172.21.15.93:0/3613350992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c440f7a60 msgr2=0x7f4c44107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:20.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c42ffd700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c440f7a60 0x7f4c44107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:20.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c437fe700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c440f7a60 0x7f4c44107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:20.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c437fe700 1 -- 172.21.15.93:0/3613350992 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c34005040 con 0x7f4c440f67c0 2021-11-20T08:56:20.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c42ffd700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c440f7a60 0x7f4c44107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T08:56:20.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.444+0000 7f4c437fe700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c441070a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f4c34009f50 tx=0x7f4c34009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:20.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.445+0000 7f4c2bfff700 1 -- 172.21.15.93:0/3613350992 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c34018070 con 0x7f4c440f67c0 2021-11-20T08:56:20.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.445+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c441085c0 con 0x7f4c440f67c0 2021-11-20T08:56:20.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.445+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c44108b20 con 0x7f4c440f67c0 2021-11-20T08:56:20.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.445+0000 7f4c2bfff700 1 -- 172.21.15.93:0/3613350992 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c34013410 con 0x7f4c440f67c0 2021-11-20T08:56:20.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.445+0000 7f4c2bfff700 1 -- 172.21.15.93:0/3613350992 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c3401c620 con 0x7f4c440f67c0 2021-11-20T08:56:20.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.446+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c44045ad0 con 0x7f4c440f67c0 2021-11-20T08:56:20.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.446+0000 7f4c2bfff700 1 -- 172.21.15.93:0/3613350992 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4c3401c780 con 0x7f4c440f67c0 2021-11-20T08:56:20.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.446+0000 7f4c2bfff700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c2c05c2a0 0x7f4c2c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:20.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.446+0000 7f4c2bfff700 1 -- 172.21.15.93:0/3613350992 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4c34077930 con 0x7f4c440f67c0 2021-11-20T08:56:20.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.446+0000 7f4c42ffd700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c2c05c2a0 0x7f4c2c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:20.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.447+0000 7f4c42ffd700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c2c05c2a0 0x7f4c2c05e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4c380041d0 tx=0x7f4c38009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:20.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.450+0000 7f4c2bfff700 1 -- 172.21.15.93:0/3613350992 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c3404e290 con 0x7f4c440f67c0 2021-11-20T08:56:20.600 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.600+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4c44027a00 con 0x7f4c440f67c0 2021-11-20T08:56:20.600 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.600+0000 7f4c2bfff700 1 -- 172.21.15.93:0/3613350992 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4c3404e4a0 con 0x7f4c440f67c0 2021-11-20T08:56:20.601 INFO:teuthology.orchestra.run.smithi093.stdout:137438953484 2021-11-20T08:56:20.602 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c2c05c2a0 msgr2=0x7f4c2c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:20.602 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c2c05c2a0 0x7f4c2c05e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4c380041d0 tx=0x7f4c38009040).stop 2021-11-20T08:56:20.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 msgr2=0x7f4c441070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:20.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c441070a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f4c34009f50 tx=0x7f4c34009b30).stop 2021-11-20T08:56:20.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 shutdown_connections 2021-11-20T08:56:20.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c2c05c2a0 0x7f4c2c05e760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:20.604 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c440f67c0 0x7f4c441070a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:20.604 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 --2- 172.21.15.93:0/3613350992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c440f7a60 0x7f4c44107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:20.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 >> 172.21.15.93:0/3613350992 conn(0x7f4c440f1f20 msgr2=0x7f4c440fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:20.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 shutdown_connections 2021-11-20T08:56:20.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:20.602+0000 7f4c49c33700 1 -- 172.21.15.93:0/3613350992 wait complete. 2021-11-20T08:56:20.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:20 smithi093 conmon[33522]: audit 2021-11-20T08:56:20.602470+0000 mon.smithi093 (mon.0) 444 2021-11-20T08:56:20.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:20 smithi093 conmon[33522]: : audit [DBG] from='client.? 172.21.15.93:0/3613350992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T08:56:20.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:20 smithi102 conmon[36472]: audit 2021-11-20T08:56:20.602470+0000 mon.smithi093 (mon.0) 444 2021-11-20T08:56:20.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:20 smithi102 conmon[36472]: : audit [DBG] from='client.? 172.21.15.93:0/3613350992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T08:56:21.004 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953482 got 137438953484 for osd.4 2021-11-20T08:56:21.005 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-11-20T08:56:21.441 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:22.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 -- 172.21.15.93:0/3705877571 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f78b0 msgr2=0x7f9ad00f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:22.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 --2- 172.21.15.93:0/3705877571 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f78b0 0x7f9ad00f7d10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9ac4004660 tx=0x7f9ac4009b30).stop 2021-11-20T08:56:22.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 -- 172.21.15.93:0/3705877571 shutdown_connections 2021-11-20T08:56:22.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 --2- 172.21.15.93:0/3705877571 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f78b0 0x7f9ad00f7d10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:22.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 --2- 172.21.15.93:0/3705877571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad00f6610 0x7f9ad00f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:22.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 -- 172.21.15.93:0/3705877571 >> 172.21.15.93:0/3705877571 conn(0x7f9ad00f1d70 msgr2=0x7f9ad00f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:22.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 -- 172.21.15.93:0/3705877571 shutdown_connections 2021-11-20T08:56:22.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.481+0000 7f9ad5761700 1 -- 172.21.15.93:0/3705877571 wait complete. 2021-11-20T08:56:22.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9ad5761700 1 Processor -- start 2021-11-20T08:56:22.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9ad5761700 1 -- start start 2021-11-20T08:56:22.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9ad5761700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f6610 0x7f9ad0106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:22.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9ad5761700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad00f78b0 0x7f9ad01073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:22.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9ad5761700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad0107970 con 0x7f9ad00f6610 2021-11-20T08:56:22.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9ad5761700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad0107ab0 con 0x7f9ad00f78b0 2021-11-20T08:56:22.491 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f6610 0x7f9ad0106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:22.491 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f6610 0x7f9ad0106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42930/0 (socket says 172.21.15.93:42930) 2021-11-20T08:56:22.492 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acffff700 1 -- 172.21.15.93:0/2160281683 learned_addr learned my addr 172.21.15.93:0/2160281683 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:22.492 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acf7fe700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad00f78b0 0x7f9ad01073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:22.493 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acffff700 1 -- 172.21.15.93:0/2160281683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad00f78b0 msgr2=0x7f9ad01073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:22.493 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acffff700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad00f78b0 0x7f9ad01073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:22.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acffff700 1 -- 172.21.15.93:0/2160281683 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac4005040 con 0x7f9ad00f6610 2021-11-20T08:56:22.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acf7fe700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad00f78b0 0x7f9ad01073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:56:22.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.482+0000 7f9acffff700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f6610 0x7f9ad0106ed0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9ac0002700 tx=0x7f9ac0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:22.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.483+0000 7f9accff9700 1 -- 172.21.15.93:0/2160281683 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ac0009320 con 0x7f9ad00f6610 2021-11-20T08:56:22.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.483+0000 7f9accff9700 1 -- 172.21.15.93:0/2160281683 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ac000f040 con 0x7f9ad00f6610 2021-11-20T08:56:22.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.483+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad0108450 con 0x7f9ad00f6610 2021-11-20T08:56:22.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.483+0000 7f9accff9700 1 -- 172.21.15.93:0/2160281683 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ac0014900 con 0x7f9ad00f6610 2021-11-20T08:56:22.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.483+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad0108a10 con 0x7f9ad00f6610 2021-11-20T08:56:22.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.484+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad0045ad0 con 0x7f9ad00f6610 2021-11-20T08:56:22.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.484+0000 7f9accff9700 1 -- 172.21.15.93:0/2160281683 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ac0009480 con 0x7f9ad00f6610 2021-11-20T08:56:22.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.484+0000 7f9accff9700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ab805c2b0 0x7f9ab805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:22.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.484+0000 7f9accff9700 1 -- 172.21.15.93:0/2160281683 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9ac0075ed0 con 0x7f9ad00f6610 2021-11-20T08:56:22.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.484+0000 7f9acf7fe700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ab805c2b0 0x7f9ab805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:22.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.485+0000 7f9acf7fe700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ab805c2b0 0x7f9ab805e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9ac4005010 tx=0x7f9ac4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:22.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.488+0000 7f9accff9700 1 -- 172.21.15.93:0/2160281683 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ac0048ed0 con 0x7f9ad00f6610 2021-11-20T08:56:22.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.641+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9ad00fef20 con 0x7f9ad00f6610 2021-11-20T08:56:22.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.641+0000 7f9accff9700 1 -- 172.21.15.93:0/2160281683 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9ac004c4f0 con 0x7f9ad00f6610 2021-11-20T08:56:22.641 INFO:teuthology.orchestra.run.smithi093.stdout:158913789962 2021-11-20T08:56:22.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ab805c2b0 msgr2=0x7f9ab805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:22.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ab805c2b0 0x7f9ab805e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9ac4005010 tx=0x7f9ac4009b30).stop 2021-11-20T08:56:22.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f6610 msgr2=0x7f9ad0106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:22.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f6610 0x7f9ad0106ed0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9ac0002700 tx=0x7f9ac0009d80).stop 2021-11-20T08:56:22.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 shutdown_connections 2021-11-20T08:56:22.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ab805c2b0 0x7f9ab805e770 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:22.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ad00f6610 0x7f9ad0106ed0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:22.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 --2- 172.21.15.93:0/2160281683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad00f78b0 0x7f9ad01073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:22.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 >> 172.21.15.93:0/2160281683 conn(0x7f9ad00f1d70 msgr2=0x7f9ad00fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:22.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 shutdown_connections 2021-11-20T08:56:22.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:22.643+0000 7f9ad5761700 1 -- 172.21.15.93:0/2160281683 wait complete. 2021-11-20T08:56:22.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:22 smithi102 conmon[36472]: cluster 2021-11-20T08:56:21.517640+0000 mgr.smithi093.qdikkm (mgr.14162) 161 : cluster 2021-11-20T08:56:22.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:22 smithi102 conmon[36472]: [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:22 smithi093 conmon[33522]: cluster 2021-11-20T08:56:21.517640+0000 mgr.smithi093.qdikkm 2021-11-20T08:56:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:22 smithi093 conmon[33522]: (mgr.14162) 161 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:23.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789960 got 158913789962 for osd.5 2021-11-20T08:56:23.148 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-11-20T08:56:23.149 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T08:56:23.583 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:23.804 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:23 smithi093 conmon[33522]: audit 2021-11-20T08:56:22.643094+0000 2021-11-20T08:56:23.805 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:23 smithi093 conmon[33522]: mon.smithi093 (mon.0) 445 : audit [DBG] from='client.? 172.21.15.93:0/2160281683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T08:56:23.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:23 smithi102 conmon[36472]: audit 2021-11-20T08:56:22.643094+0000 mon.smithi093 (mon 2021-11-20T08:56:23.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:23 smithi102 conmon[36472]: .0) 445 : audit [DBG] from='client.? 172.21.15.93:0/2160281683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T08:56:24.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 -- 172.21.15.93:0/4017131913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 msgr2=0x7f17e40f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:24.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 --2- 172.21.15.93:0/4017131913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40f6ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f17d4004660 tx=0x7f17d4009b30).stop 2021-11-20T08:56:24.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 -- 172.21.15.93:0/4017131913 shutdown_connections 2021-11-20T08:56:24.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 --2- 172.21.15.93:0/4017131913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17e40f7390 0x7f17e40f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:24.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 --2- 172.21.15.93:0/4017131913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40f6ea0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:24.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 -- 172.21.15.93:0/4017131913 >> 172.21.15.93:0/4017131913 conn(0x7f17e40f0430 msgr2=0x7f17e40f2850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:24.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 -- 172.21.15.93:0/4017131913 shutdown_connections 2021-11-20T08:56:24.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.680+0000 7f17e88ba700 1 -- 172.21.15.93:0/4017131913 wait complete. 2021-11-20T08:56:24.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e88ba700 1 Processor -- start 2021-11-20T08:56:24.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e88ba700 1 -- start start 2021-11-20T08:56:24.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e88ba700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40fd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:24.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e88ba700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17e40f7390 0x7f17e40fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:24.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e88ba700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40fc0f0 con 0x7f17e40f6a80 2021-11-20T08:56:24.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e88ba700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40fc230 con 0x7f17e40f7390 2021-11-20T08:56:24.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:24.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40fd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42940/0 (socket says 172.21.15.93:42940) 2021-11-20T08:56:24.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e37fe700 1 -- 172.21.15.93:0/189483985 learned_addr learned my addr 172.21.15.93:0/189483985 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:24.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.681+0000 7f17e2ffd700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17e40f7390 0x7f17e40fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:24.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17e37fe700 1 -- 172.21.15.93:0/189483985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17e40f7390 msgr2=0x7f17e40fbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:24.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17e37fe700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17e40f7390 0x7f17e40fbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:24.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17e37fe700 1 -- 172.21.15.93:0/189483985 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d4005040 con 0x7f17e40f6a80 2021-11-20T08:56:24.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17e37fe700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40fd710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f17d4004e30 tx=0x7f17d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:24.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17cbfff700 1 -- 172.21.15.93:0/189483985 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d4018070 con 0x7f17e40f6a80 2021-11-20T08:56:24.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e40fc4b0 con 0x7f17e40f6a80 2021-11-20T08:56:24.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e40fca10 con 0x7f17e40f6a80 2021-11-20T08:56:24.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17cbfff700 1 -- 172.21.15.93:0/189483985 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17d4013410 con 0x7f17e40f6a80 2021-11-20T08:56:24.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.682+0000 7f17cbfff700 1 -- 172.21.15.93:0/189483985 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d401dc40 con 0x7f17e40f6a80 2021-11-20T08:56:24.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.683+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c4004fa0 con 0x7f17e40f6a80 2021-11-20T08:56:24.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.684+0000 7f17cbfff700 1 -- 172.21.15.93:0/189483985 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f17d401d420 con 0x7f17e40f6a80 2021-11-20T08:56:24.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.684+0000 7f17cbfff700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f17cc05c300 0x7f17cc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:24.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.684+0000 7f17cbfff700 1 -- 172.21.15.93:0/189483985 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f17d4077950 con 0x7f17e40f6a80 2021-11-20T08:56:24.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.684+0000 7f17e2ffd700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f17cc05c300 0x7f17cc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:24.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.684+0000 7f17e2ffd700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f17cc05c300 0x7f17cc05e7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f17d8004e20 tx=0x7f17d800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:24.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.687+0000 7f17cbfff700 1 -- 172.21.15.93:0/189483985 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17d404a9d0 con 0x7f17e40f6a80 2021-11-20T08:56:24.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:24 smithi093 conmon[33522]: cluster 2021-11-20T08:56:23.518088+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:56:24.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:24 smithi093 conmon[33522]: 14162) 162 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:24.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.835+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17c4000bc0 con 0x7f17cc05c300 2021-11-20T08:56:24.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.836+0000 7f17cbfff700 1 -- 172.21.15.93:0/189483985 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24060 (secure 0 0 0) 0x7f17c4000bc0 con 0x7f17cc05c300 2021-11-20T08:56:24.837 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:24.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f17cc05c300 msgr2=0x7f17cc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:24.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f17cc05c300 0x7f17cc05e7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f17d8004e20 tx=0x7f17d800b040).stop 2021-11-20T08:56:24.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 msgr2=0x7f17e40fd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:24.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40fd710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f17d4004e30 tx=0x7f17d4009b30).stop 2021-11-20T08:56:24.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 shutdown_connections 2021-11-20T08:56:24.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f17cc05c300 0x7f17cc05e7c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:24.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17e40f6a80 0x7f17e40fd710 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:24.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 --2- 172.21.15.93:0/189483985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17e40f7390 0x7f17e40fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:24.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 >> 172.21.15.93:0/189483985 conn(0x7f17e40f0430 msgr2=0x7f17e40fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:24.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 shutdown_connections 2021-11-20T08:56:24.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:24.838+0000 7f17e88ba700 1 -- 172.21.15.93:0/189483985 wait complete. 2021-11-20T08:56:24.841 INFO:teuthology.orchestra.run.smithi093.stderr:dumped all 2021-11-20T08:56:24.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:24 smithi102 conmon[36472]: cluster 2021-11-20T08:56:23.518088+0000 mgr.smithi093.qdikkm (mgr.14162) 162 2021-11-20T08:56:24.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:24 smithi102 conmon[36472]: : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:25.278 INFO:teuthology.orchestra.run.smithi093.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2021-11-20T08:56:23.517871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31224,"kb_used_data":1992,"kb_used_omap":0,"kb_used_meta":29184,"kb_avail":562439688,"statfs":{"total":575970213888,"available":575938240512,"internally_reserved":0,"allocated":2039808,"data_stored":764922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29884416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"37","state":"active+clean","last_fresh":"2021-11-20T08:55:49.154872+0000","last_change":"2021-11-20T08:55:32.253246+0000","last_active":"2021-11-20T08:55:49.154872+0000","last_peered":"2021-11-20T08:55:49.154872+0000","last_clean":"2021-11-20T08:55:49.154872+0000","last_became_active":"2021-11-20T08:55:32.252858+0000","last_became_peered":"2021-11-20T08:55:32.252858+0000","last_unstale":"2021-11-20T08:55:49.154872+0000","last_undegraded":"2021-11-20T08:55:49.154872+0000","last_fullsized":"2021-11-20T08:55:49.154872+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T08:55:09.864699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T08:55:09.864699+0000","last_clean_scrub_stamp":"2021-11-20T08:55:09.864699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":37,"seq":158913789962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5076,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740076,"statfs":{"total":95995035648,"available":95989837824,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]},{"osd":4,"up_from":32,"seq":137438953485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5140,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740012,"statfs":{"total":95995035648,"available":95989772288,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5204,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739948,"statfs":{"total":95995035648,"available":95989706752,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.315},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.376},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5204,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739948,"statfs":{"total":95995035648,"available":95989706752,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 08:56:05 2021","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.25600000000000001}]},{"osd":1,"last update":"Sat Nov 20 08:56:05 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.307},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.498,"5min":0.498,"15min":0.498},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5332,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739820,"statfs":{"total":95995035648,"available":95989575680,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 08:55:43 2021","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.28299999999999997}]},{"osd":2,"last update":"Sat Nov 20 08:56:01 2021","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.32100000000000001}]},{"osd":3,"last update":"Sat Nov 20 08:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.621},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.64900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5268,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739884,"statfs":{"total":95995035648,"available":95989641216,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 08:55:44 2021","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.59899999999999998}]},{"osd":2,"last update":"Sat Nov 20 08:55:59 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.624},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.61199999999999999}]},{"osd":3,"last update":"Sat Nov 20 08:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.86099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T08:56:25.280 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T08:56:25.714 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:25.862 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:25 smithi093 conmon[33522]: audit 2021-11-20T08:56:24.837593+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:56:25.862 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:25 smithi093 conmon[33522]: .14162) 163 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T08:56:25.862 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:25 smithi093 conmon[33522]: cluster 2021-11-20T08:56:25.518601+0000 mgr.smithi093.qdikkm (mgr.14162) 164 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:25.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:25 smithi102 conmon[36472]: audit 2021-11-20T08:56:24.837593+0000 mgr.smithi093.qdikkm (mgr.14162) 163 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T08:56:25.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:25 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T08:56:25.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:25 smithi102 conmon[36472]: 08:56:25.518601+0000 mgr.smithi093.qdikkm (mgr.14162) 164 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:26.926 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 -- 172.21.15.93:0/237549412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 msgr2=0x7f63600f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:26.926 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 --2- 172.21.15.93:0/237549412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f63600f6de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f634c004660 tx=0x7f634c009b30).stop 2021-11-20T08:56:26.927 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 -- 172.21.15.93:0/237549412 shutdown_connections 2021-11-20T08:56:26.927 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 --2- 172.21.15.93:0/237549412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f63600f6de0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:26.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 --2- 172.21.15.93:0/237549412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63600f56e0 0x7f63600f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:26.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 -- 172.21.15.93:0/237549412 >> 172.21.15.93:0/237549412 conn(0x7f63600f0e20 msgr2=0x7f63600f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:26.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 -- 172.21.15.93:0/237549412 shutdown_connections 2021-11-20T08:56:26.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 -- 172.21.15.93:0/237549412 wait complete. 2021-11-20T08:56:26.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 Processor -- start 2021-11-20T08:56:26.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.922+0000 7f6366e1c700 1 -- start start 2021-11-20T08:56:26.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f6366e1c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63600f56e0 0x7f636010c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:26.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f6366e1c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f636010e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f6366e1c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6360060830 con 0x7f63600f6980 2021-11-20T08:56:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f6366e1c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6360060970 con 0x7f63600f56e0 2021-11-20T08:56:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f636010e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:26.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f636010e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42952/0 (socket says 172.21.15.93:42952) 2021-11-20T08:56:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635ffff700 1 -- 172.21.15.93:0/2557766407 learned_addr learned my addr 172.21.15.93:0/2557766407 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f6364bb8700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63600f56e0 0x7f636010c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635ffff700 1 -- 172.21.15.93:0/2557766407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63600f56e0 msgr2=0x7f636010c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:26.931 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635ffff700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63600f56e0 0x7f636010c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635ffff700 1 -- 172.21.15.93:0/2557766407 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c005040 con 0x7f63600f6980 2021-11-20T08:56:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635ffff700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f636010e650 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f634c000c00 tx=0x7f634c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f635d7fa700 1 -- 172.21.15.93:0/2557766407 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634c019070 con 0x7f63600f6980 2021-11-20T08:56:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.923+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636010eb40 con 0x7f63600f6980 2021-11-20T08:56:26.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.924+0000 7f635d7fa700 1 -- 172.21.15.93:0/2557766407 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f634c007e70 con 0x7f63600f6980 2021-11-20T08:56:26.933 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.924+0000 7f635d7fa700 1 -- 172.21.15.93:0/2557766407 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634c014d00 con 0x7f63600f6980 2021-11-20T08:56:26.933 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.924+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636010ef10 con 0x7f63600f6980 2021-11-20T08:56:26.933 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.925+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6360045ad0 con 0x7f63600f6980 2021-11-20T08:56:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.925+0000 7f635d7fa700 1 -- 172.21.15.93:0/2557766407 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f634c025070 con 0x7f63600f6980 2021-11-20T08:56:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.925+0000 7f635d7fa700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f635005c2a0 0x7f635005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.925+0000 7f635d7fa700 1 -- 172.21.15.93:0/2557766407 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f634c076bf0 con 0x7f63600f6980 2021-11-20T08:56:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.925+0000 7f6364bb8700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f635005c2a0 0x7f635005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:26.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.926+0000 7f6364bb8700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f635005c2a0 0x7f635005e760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f63600605c0 tx=0x7f635400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:26.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:26.929+0000 7f635d7fa700 1 -- 172.21.15.93:0/2557766407 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f634c049b90 con 0x7f63600f6980 2021-11-20T08:56:27.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.076+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6360057f10 con 0x7f635005c2a0 2021-11-20T08:56:27.078 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.077+0000 7f635d7fa700 1 -- 172.21.15.93:0/2557766407 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24074 (secure 0 0 0) 0x7f6360057f10 con 0x7f635005c2a0 2021-11-20T08:56:27.079 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:27.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.078+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f635005c2a0 msgr2=0x7f635005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:27.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.078+0000 7f6366e1c700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f635005c2a0 0x7f635005e760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f63600605c0 tx=0x7f635400b040).stop 2021-11-20T08:56:27.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.078+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 msgr2=0x7f636010e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:27.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.078+0000 7f6366e1c700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f636010e650 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f634c000c00 tx=0x7f634c00b040).stop 2021-11-20T08:56:27.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.079+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 shutdown_connections 2021-11-20T08:56:27.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.079+0000 7f6366e1c700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f635005c2a0 0x7f635005e760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:27.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.079+0000 7f6366e1c700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63600f56e0 0x7f636010c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:27.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.079+0000 7f6366e1c700 1 --2- 172.21.15.93:0/2557766407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63600f6980 0x7f636010e650 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:27.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.079+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 >> 172.21.15.93:0/2557766407 conn(0x7f63600f0e20 msgr2=0x7f63600fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:27.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.079+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 shutdown_connections 2021-11-20T08:56:27.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:27.079+0000 7f6366e1c700 1 -- 172.21.15.93:0/2557766407 wait complete. 2021-11-20T08:56:27.083 INFO:teuthology.orchestra.run.smithi093.stderr:dumped all 2021-11-20T08:56:27.411 INFO:teuthology.orchestra.run.smithi093.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2021-11-20T08:56:25.518366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31224,"kb_used_data":1992,"kb_used_omap":0,"kb_used_meta":29184,"kb_avail":562439688,"statfs":{"total":575970213888,"available":575938240512,"internally_reserved":0,"allocated":2039808,"data_stored":764922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29884416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"37","state":"active+clean","last_fresh":"2021-11-20T08:55:49.154872+0000","last_change":"2021-11-20T08:55:32.253246+0000","last_active":"2021-11-20T08:55:49.154872+0000","last_peered":"2021-11-20T08:55:49.154872+0000","last_clean":"2021-11-20T08:55:49.154872+0000","last_became_active":"2021-11-20T08:55:32.252858+0000","last_became_peered":"2021-11-20T08:55:32.252858+0000","last_unstale":"2021-11-20T08:55:49.154872+0000","last_undegraded":"2021-11-20T08:55:49.154872+0000","last_fullsized":"2021-11-20T08:55:49.154872+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T08:55:09.864699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T08:55:09.864699+0000","last_clean_scrub_stamp":"2021-11-20T08:55:09.864699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":37,"seq":158913789963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5076,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740076,"statfs":{"total":95995035648,"available":95989837824,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5140,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740012,"statfs":{"total":95995035648,"available":95989772288,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5204,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739948,"statfs":{"total":95995035648,"available":95989706752,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.315},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.376},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378645,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5204,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739948,"statfs":{"total":95995035648,"available":95989706752,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 08:56:05 2021","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.33500000000000002}]},{"osd":1,"last update":"Sat Nov 20 08:56:05 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.498,"5min":0.498,"15min":0.498},"last":0.315}]},{"osd":3,"last update":"Sat Nov 20 08:56:15 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5332,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739820,"statfs":{"total":95995035648,"available":95989575680,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 08:55:43 2021","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Sat Nov 20 08:56:01 2021","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.31900000000000001}]},{"osd":3,"last update":"Sat Nov 20 08:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5268,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739884,"statfs":{"total":95995035648,"available":95989641216,"internally_reserved":0,"allocated":339968,"data_stored":127487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 08:55:44 2021","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.249}]},{"osd":2,"last update":"Sat Nov 20 08:55:59 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.311}]},{"osd":3,"last update":"Sat Nov 20 08:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T08:56:27.413 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-11-20T08:56:27.413 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-11-20T08:56:27.413 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-11-20T08:56:27.414 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- ceph health --format=json 2021-11-20T08:56:27.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:27 smithi093 conmon[33522]: audit 2021-11-20T08:56:27.078176+0000 mgr.smithi093.qdikkm (mgr.14162) 165 : audit [DBG] from='client.14386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T08:56:27.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:27 smithi093 conmon[33522]: cluster 2021-11-20T08:56:27.518993+0000 mgr.smithi093.qdikkm (mgr.14162) 166 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:27.856 INFO:teuthology.orchestra.run.smithi093.stderr:Inferring config /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/mon.smithi093/config 2021-11-20T08:56:27.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:27 smithi102 conmon[36472]: audit 2021-11-20T08:56:27.078176+0000 mgr.smithi093.qdikkm (mgr.14162) 165 : audit [DBG] from='client.14386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T08:56:27.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:27 smithi102 conmon[36472]: cluster 2021-11-20T08:56:27.518993+0000 mgr.smithi093.qdikkm 2021-11-20T08:56:27.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:27 smithi102 conmon[36472]: (mgr.14162) 166 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:29.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 -- 172.21.15.93:0/3545318291 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 msgr2=0x7f2fa40f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:29.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 --2- 172.21.15.93:0/3545318291 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa40f6be0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2f98004660 tx=0x7f2f98009b30).stop 2021-11-20T08:56:29.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 -- 172.21.15.93:0/3545318291 shutdown_connections 2021-11-20T08:56:29.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 --2- 172.21.15.93:0/3545318291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa40f7a60 0x7f2fa40f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:29.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 --2- 172.21.15.93:0/3545318291 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa40f6be0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:29.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 -- 172.21.15.93:0/3545318291 >> 172.21.15.93:0/3545318291 conn(0x7f2fa40f1f20 msgr2=0x7f2fa40f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:29.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 -- 172.21.15.93:0/3545318291 shutdown_connections 2021-11-20T08:56:29.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 -- 172.21.15.93:0/3545318291 wait complete. 2021-11-20T08:56:29.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 Processor -- start 2021-11-20T08:56:29.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.016+0000 7f2faad73700 1 -- start start 2021-11-20T08:56:29.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2faad73700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:29.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2faad73700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa40f7a60 0x7f2fa4069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:29.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2faad73700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa406a380 con 0x7f2fa40f67c0 2021-11-20T08:56:29.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2faad73700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa406a4c0 con 0x7f2fa40f7a60 2021-11-20T08:56:29.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa8b0f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:29.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa8b0f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42960/0 (socket says 172.21.15.93:42960) 2021-11-20T08:56:29.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa8b0f700 1 -- 172.21.15.93:0/1680327436 learned_addr learned my addr 172.21.15.93:0/1680327436 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:29.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa3fff700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa40f7a60 0x7f2fa4069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:29.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa8b0f700 1 -- 172.21.15.93:0/1680327436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa40f7a60 msgr2=0x7f2fa4069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:29.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa8b0f700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa40f7a60 0x7f2fa4069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:29.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa8b0f700 1 -- 172.21.15.93:0/1680327436 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f98005040 con 0x7f2fa40f67c0 2021-11-20T08:56:29.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa8b0f700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa406b9a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f2f98004e30 tx=0x7f2f98009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:29.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2fa17fa700 1 -- 172.21.15.93:0/1680327436 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f98018070 con 0x7f2fa40f67c0 2021-11-20T08:56:29.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.017+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa406a740 con 0x7f2fa40f67c0 2021-11-20T08:56:29.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.018+0000 7f2fa17fa700 1 -- 172.21.15.93:0/1680327436 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f98013410 con 0x7f2fa40f67c0 2021-11-20T08:56:29.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.018+0000 7f2fa17fa700 1 -- 172.21.15.93:0/1680327436 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f9801c3c0 con 0x7f2fa40f67c0 2021-11-20T08:56:29.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.018+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa406aca0 con 0x7f2fa40f67c0 2021-11-20T08:56:29.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.018+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa4045ad0 con 0x7f2fa40f67c0 2021-11-20T08:56:29.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.019+0000 7f2fa17fa700 1 -- 172.21.15.93:0/1680327436 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2f9801d420 con 0x7f2fa40f67c0 2021-11-20T08:56:29.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.019+0000 7f2fa17fa700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2f9405c2a0 0x7f2f9405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:29.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.019+0000 7f2fa17fa700 1 -- 172.21.15.93:0/1680327436 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2f980777d0 con 0x7f2fa40f67c0 2021-11-20T08:56:29.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.019+0000 7f2fa3fff700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2f9405c2a0 0x7f2f9405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:29.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.020+0000 7f2fa3fff700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2f9405c2a0 0x7f2f9405e760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2f900077b0 tx=0x7f2f90009ce0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:29.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.023+0000 7f2fa17fa700 1 -- 172.21.15.93:0/1680327436 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f9804a7d0 con 0x7f2fa40f67c0 2021-11-20T08:56:29.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.230+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2fa4027a00 con 0x7f2fa40f67c0 2021-11-20T08:56:29.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.230+0000 7f2fa17fa700 1 -- 172.21.15.93:0/1680327436 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f2f9804ddf0 con 0x7f2fa40f67c0 2021-11-20T08:56:29.230 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:29.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2f9405c2a0 msgr2=0x7f2f9405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:29.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2f9405c2a0 0x7f2f9405e760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2f900077b0 tx=0x7f2f90009ce0).stop 2021-11-20T08:56:29.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 msgr2=0x7f2fa406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:29.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa406b9a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f2f98004e30 tx=0x7f2f98009b30).stop 2021-11-20T08:56:29.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 shutdown_connections 2021-11-20T08:56:29.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2f9405c2a0 0x7f2f9405e760 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:29.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2fa40f67c0 0x7f2fa406b9a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:29.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 --2- 172.21.15.93:0/1680327436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa40f7a60 0x7f2fa4069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:29.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 >> 172.21.15.93:0/1680327436 conn(0x7f2fa40f1f20 msgr2=0x7f2fa40fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:29.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 shutdown_connections 2021-11-20T08:56:29.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:29.232+0000 7f2faad73700 1 -- 172.21.15.93:0/1680327436 wait complete. 2021-11-20T08:56:29.590 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:29 smithi093 conmon[33522]: audit 2021-11-20T08:56:29.231578+0000 mon.smithi093 (mon.0) 446 : audit [DBG] from='client.? 172.21.15.93:0/1680327436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T08:56:29.653 INFO:teuthology.orchestra.run.smithi093.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-11-20T08:56:29.654 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-11-20T08:56:29.654 INFO:tasks.cephadm:Setup complete, yielding 2021-11-20T08:56:29.655 INFO:teuthology.run_tasks:Running task print... 2021-11-20T08:56:29.666 INFO:teuthology.task.print:**** done starting v16.2.4 2021-11-20T08:56:29.667 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T08:56:29.705 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T08:56:29.707 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch status' 2021-11-20T08:56:29.708 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:29 smithi102 conmon[36472]: audit 2021-11-20T08:56:29.231578+0000 mon.smithi093 (mon.0) 446 : audit [DBG] from='client.? 172.21.15.93:0/1680327436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T08:56:30.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:30 smithi093 conmon[33522]: cluster 2021-11-20T08:56:29.519320+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:56:30.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:30 smithi093 conmon[33522]: ) 167 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:30.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:30 smithi102 conmon[36472]: cluster 2021-11-20T08:56:29.519320+0000 mgr.smithi093.qdikkm (mgr.14162) 167 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:31.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 -- 172.21.15.93:0/1805838307 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 msgr2=0x7f30900f5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:31.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 --2- 172.21.15.93:0/1805838307 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f30900f5800 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3084004660 tx=0x7f3084009b30).stop 2021-11-20T08:56:31.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 -- 172.21.15.93:0/1805838307 shutdown_connections 2021-11-20T08:56:31.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 --2- 172.21.15.93:0/1805838307 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f30900f5800 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:31.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 --2- 172.21.15.93:0/1805838307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30900f6a40 0x7f30900f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:31.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 -- 172.21.15.93:0/1805838307 >> 172.21.15.93:0/1805838307 conn(0x7f30900f0330 msgr2=0x7f30900f2750 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:31.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 -- 172.21.15.93:0/1805838307 shutdown_connections 2021-11-20T08:56:31.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.163+0000 7f308ffff700 1 -- 172.21.15.93:0/1805838307 wait complete. 2021-11-20T08:56:31.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308ffff700 1 Processor -- start 2021-11-20T08:56:31.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308ffff700 1 -- start start 2021-11-20T08:56:31.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30900f6a40 0x7f309010bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:31.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f309010e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:31.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308ffff700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30900f5360 con 0x7f30900f73e0 2021-11-20T08:56:31.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308ffff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30900f54a0 con 0x7f30900f6a40 2021-11-20T08:56:31.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308e7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f309010e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:31.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308e7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f309010e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42970/0 (socket says 172.21.15.93:42970) 2021-11-20T08:56:31.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308e7fc700 1 -- 172.21.15.93:0/3899754455 learned_addr learned my addr 172.21.15.93:0/3899754455 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:31.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.164+0000 7f308effd700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30900f6a40 0x7f309010bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:31.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f308e7fc700 1 -- 172.21.15.93:0/3899754455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30900f6a40 msgr2=0x7f309010bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:31.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f308e7fc700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30900f6a40 0x7f309010bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:31.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f308e7fc700 1 -- 172.21.15.93:0/3899754455 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084005040 con 0x7f30900f73e0 2021-11-20T08:56:31.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f308e7fc700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f309010e3a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3084009830 tx=0x7f3084009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:31.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f30777fe700 1 -- 172.21.15.93:0/3899754455 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3084018070 con 0x7f30900f73e0 2021-11-20T08:56:31.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309010e890 con 0x7f30900f73e0 2021-11-20T08:56:31.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f30777fe700 1 -- 172.21.15.93:0/3899754455 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3084013410 con 0x7f30900f73e0 2021-11-20T08:56:31.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f30777fe700 1 -- 172.21.15.93:0/3899754455 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308401c400 con 0x7f30900f73e0 2021-11-20T08:56:31.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.165+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309010ecb0 con 0x7f30900f73e0 2021-11-20T08:56:31.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.166+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3090045ad0 con 0x7f30900f73e0 2021-11-20T08:56:31.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.166+0000 7f30777fe700 1 -- 172.21.15.93:0/3899754455 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f308401d420 con 0x7f30900f73e0 2021-11-20T08:56:31.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.167+0000 7f30777fe700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f307805c300 0x7f307805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:31.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.167+0000 7f30777fe700 1 -- 172.21.15.93:0/3899754455 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3084077a30 con 0x7f30900f73e0 2021-11-20T08:56:31.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.167+0000 7f308effd700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f307805c300 0x7f307805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:31.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.167+0000 7f308effd700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f307805c300 0x7f307805e7c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f30900f50f0 tx=0x7f3080008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:31.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.171+0000 7f30777fe700 1 -- 172.21.15.93:0/3899754455 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f308404aa30 con 0x7f30900f73e0 2021-11-20T08:56:31.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.352+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30900fdd50 con 0x7f307805c300 2021-11-20T08:56:31.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.353+0000 7f30777fe700 1 -- 172.21.15.93:0/3899754455 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f30900fdd50 con 0x7f307805c300 2021-11-20T08:56:31.352 INFO:teuthology.orchestra.run.smithi093.stdout:Backend: cephadm 2021-11-20T08:56:31.352 INFO:teuthology.orchestra.run.smithi093.stdout:Available: Yes 2021-11-20T08:56:31.353 INFO:teuthology.orchestra.run.smithi093.stdout:Paused: No 2021-11-20T08:56:31.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f307805c300 msgr2=0x7f307805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:31.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f307805c300 0x7f307805e7c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f30900f50f0 tx=0x7f3080008040).stop 2021-11-20T08:56:31.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 msgr2=0x7f309010e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:31.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f309010e3a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3084009830 tx=0x7f3084009b30).stop 2021-11-20T08:56:31.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 shutdown_connections 2021-11-20T08:56:31.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f307805c300 0x7f307805e7c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:31.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30900f6a40 0x7f309010bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:31.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 --2- 172.21.15.93:0/3899754455 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30900f73e0 0x7f309010e3a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:31.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.354+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 >> 172.21.15.93:0/3899754455 conn(0x7f30900f0330 msgr2=0x7f30900fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:31.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.355+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 shutdown_connections 2021-11-20T08:56:31.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:31.355+0000 7f308ffff700 1 -- 172.21.15.93:0/3899754455 wait complete. 2021-11-20T08:56:31.710 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ps' 2021-11-20T08:56:32.905 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:32 smithi093 conmon[33522]: audit 2021-11-20T08:56:31.353725+0000 mgr.smithi093.qdikkm 2021-11-20T08:56:32.905 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:32 smithi093 conmon[33522]: (mgr.14162) 168 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:32.906 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:32 smithi093 conmon[33522]: cluster 2021-11-20T08:56:31.519624+0000 mgr.smithi093.qdikkm (mgr.14162) 169 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:32.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:32 smithi102 conmon[36472]: audit 2021-11-20T08:56:31.353725+0000 mgr.smithi093.qdikkm (mgr.14162) 168 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:32.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:32 smithi102 conmon[36472]: cluster 2021-11-20T08:56:31.519624+0000 mgr.smithi093.qdikkm (mgr.14162) 169 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:33.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 -- 172.21.15.93:0/539453905 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95080f96b0 msgr2=0x7f95080fbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:33.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 --2- 172.21.15.93:0/539453905 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95080f96b0 0x7f95080fbb00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f94f8004660 tx=0x7f94f8009b30).stop 2021-11-20T08:56:33.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 -- 172.21.15.93:0/539453905 shutdown_connections 2021-11-20T08:56:33.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 --2- 172.21.15.93:0/539453905 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95080f96b0 0x7f95080fbb00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:33.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 --2- 172.21.15.93:0/539453905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9508060290 0x7f95080f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:33.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 -- 172.21.15.93:0/539453905 >> 172.21.15.93:0/539453905 conn(0x7f95080f0ba0 msgr2=0x7f95080f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:33.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 -- 172.21.15.93:0/539453905 shutdown_connections 2021-11-20T08:56:33.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.130+0000 7f950c9a4700 1 -- 172.21.15.93:0/539453905 wait complete. 2021-11-20T08:56:33.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f950c9a4700 1 Processor -- start 2021-11-20T08:56:33.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f950c9a4700 1 -- start start 2021-11-20T08:56:33.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f950c9a4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9508060290 0x7f9508106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:33.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f950c9a4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95080f96b0 0x7f95081072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:33.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f950c9a4700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9508107980 con 0x7f9508060290 2021-11-20T08:56:33.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f950c9a4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9508108100 con 0x7f95080f96b0 2021-11-20T08:56:33.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f95077fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9508060290 0x7f9508106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:33.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f95077fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9508060290 0x7f9508106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42978/0 (socket says 172.21.15.93:42978) 2021-11-20T08:56:33.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.131+0000 7f95077fe700 1 -- 172.21.15.93:0/1315381638 learned_addr learned my addr 172.21.15.93:0/1315381638 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:33.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f9506ffd700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95080f96b0 0x7f95081072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:33.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f95077fe700 1 -- 172.21.15.93:0/1315381638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95080f96b0 msgr2=0x7f95081072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:33.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f95077fe700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95080f96b0 0x7f95081072f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:33.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f95077fe700 1 -- 172.21.15.93:0/1315381638 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8005040 con 0x7f9508060290 2021-11-20T08:56:33.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f95077fe700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9508060290 0x7f9508106e00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f94fc00b410 tx=0x7f94fc006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:33.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f94effff700 1 -- 172.21.15.93:0/1315381638 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94fc0125f0 con 0x7f9508060290 2021-11-20T08:56:33.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f94effff700 1 -- 172.21.15.93:0/1315381638 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94fc012c00 con 0x7f9508060290 2021-11-20T08:56:33.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9508108380 con 0x7f9508060290 2021-11-20T08:56:33.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f94effff700 1 -- 172.21.15.93:0/1315381638 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94fc002aa0 con 0x7f9508060290 2021-11-20T08:56:33.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.132+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9508108940 con 0x7f9508060290 2021-11-20T08:56:33.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.133+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9508045ad0 con 0x7f9508060290 2021-11-20T08:56:33.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.133+0000 7f94effff700 1 -- 172.21.15.93:0/1315381638 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f94fc00f030 con 0x7f9508060290 2021-11-20T08:56:33.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.134+0000 7f94effff700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f94f005c300 0x7f94f005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:33.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.134+0000 7f94effff700 1 -- 172.21.15.93:0/1315381638 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f94fc076060 con 0x7f9508060290 2021-11-20T08:56:33.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.134+0000 7f9506ffd700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f94f005c300 0x7f94f005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:33.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.134+0000 7f9506ffd700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f94f005c300 0x7f94f005e7c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f94f8004e30 tx=0x7f94f8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:33.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.138+0000 7f94effff700 1 -- 172.21.15.93:0/1315381638 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94fc049060 con 0x7f9508060290 2021-11-20T08:56:33.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.315+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f95080fef10 con 0x7f94f005c300 2021-11-20T08:56:33.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.322+0000 7f94effff700 1 -- 172.21.15.93:0/1315381638 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2160 (secure 0 0 0) 0x7f94f8019040 con 0x7f94f005c300 2021-11-20T08:56:33.322 INFO:teuthology.orchestra.run.smithi093.stdout:NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID 2021-11-20T08:56:33.322 INFO:teuthology.orchestra.run.smithi093.stdout:alertmanager.smithi093 smithi093 *:9093,9094 running (2m) 38s ago 3m 0.20.0 0881eb8f169f d7d47c19ba4d 2021-11-20T08:56:33.322 INFO:teuthology.orchestra.run.smithi093.stdout:crash.smithi093 smithi093 running (3m) 38s ago 3m 16.2.4 8d91d370c2b8 7a1ef345408e 2021-11-20T08:56:33.323 INFO:teuthology.orchestra.run.smithi093.stdout:crash.smithi102 smithi102 running (2m) 48s ago 2m 16.2.4 8d91d370c2b8 725e063e9b50 2021-11-20T08:56:33.323 INFO:teuthology.orchestra.run.smithi093.stdout:grafana.smithi093 smithi093 *:3000 running (2m) 38s ago 3m 6.7.4 557c83e11646 4e96bc3b9864 2021-11-20T08:56:33.323 INFO:teuthology.orchestra.run.smithi093.stdout:mgr.smithi093.qdikkm smithi093 *:9283 running (4m) 38s ago 4m 16.2.4 8d91d370c2b8 4b5edcffdbae 2021-11-20T08:56:33.323 INFO:teuthology.orchestra.run.smithi093.stdout:mgr.smithi102.ievjft smithi102 *:8443,9283 running (2m) 48s ago 2m 16.2.4 8d91d370c2b8 4a82da9f36a0 2021-11-20T08:56:33.324 INFO:teuthology.orchestra.run.smithi093.stdout:mon.smithi093 smithi093 running (4m) 38s ago 4m 16.2.4 8d91d370c2b8 b1085d96938e 2021-11-20T08:56:33.324 INFO:teuthology.orchestra.run.smithi093.stdout:mon.smithi102 smithi102 running (2m) 48s ago 2m 16.2.4 8d91d370c2b8 fba614458a87 2021-11-20T08:56:33.324 INFO:teuthology.orchestra.run.smithi093.stdout:node-exporter.smithi093 smithi093 *:9100 running (3m) 38s ago 3m 0.18.1 e5a616e4b9cf 59d2611057e1 2021-11-20T08:56:33.324 INFO:teuthology.orchestra.run.smithi093.stdout:node-exporter.smithi102 smithi102 *:9100 running (2m) 48s ago 2m 0.18.1 e5a616e4b9cf d615a3497bc0 2021-11-20T08:56:33.324 INFO:teuthology.orchestra.run.smithi093.stdout:osd.0 smithi093 running (2m) 38s ago 2m 16.2.4 8d91d370c2b8 9787f5dd5649 2021-11-20T08:56:33.325 INFO:teuthology.orchestra.run.smithi093.stdout:osd.1 smithi093 running (117s) 38s ago 117s 16.2.4 8d91d370c2b8 4c625443cd7d 2021-11-20T08:56:33.325 INFO:teuthology.orchestra.run.smithi093.stdout:osd.2 smithi093 running (100s) 38s ago 100s 16.2.4 8d91d370c2b8 224be14e8bf6 2021-11-20T08:56:33.325 INFO:teuthology.orchestra.run.smithi093.stdout:osd.3 smithi102 running (84s) 48s ago 84s 16.2.4 8d91d370c2b8 cd29f7548910 2021-11-20T08:56:33.326 INFO:teuthology.orchestra.run.smithi093.stdout:osd.4 smithi102 running (68s) 48s ago 68s 16.2.4 8d91d370c2b8 a05e242dadc0 2021-11-20T08:56:33.327 INFO:teuthology.orchestra.run.smithi093.stdout:osd.5 smithi102 running (51s) 48s ago 50s 16.2.4 8d91d370c2b8 dd15bcc9c8f6 2021-11-20T08:56:33.327 INFO:teuthology.orchestra.run.smithi093.stdout:prometheus.smithi093 smithi093 *:9095 running (2m) 38s ago 3m 2.18.1 de242295e225 71ec953e81f4 2021-11-20T08:56:33.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.323+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f94f005c300 msgr2=0x7f94f005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:33.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.323+0000 7f950c9a4700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f94f005c300 0x7f94f005e7c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f94f8004e30 tx=0x7f94f8009b30).stop 2021-11-20T08:56:33.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.323+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9508060290 msgr2=0x7f9508106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:33.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.323+0000 7f950c9a4700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9508060290 0x7f9508106e00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f94fc00b410 tx=0x7f94fc006ef0).stop 2021-11-20T08:56:33.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.324+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 shutdown_connections 2021-11-20T08:56:33.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.324+0000 7f950c9a4700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f94f005c300 0x7f94f005e7c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:33.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.324+0000 7f950c9a4700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9508060290 0x7f9508106e00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:33.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.324+0000 7f950c9a4700 1 --2- 172.21.15.93:0/1315381638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95080f96b0 0x7f95081072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:33.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.324+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 >> 172.21.15.93:0/1315381638 conn(0x7f95080f0ba0 msgr2=0x7f95080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:33.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.324+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 shutdown_connections 2021-11-20T08:56:33.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:33.324+0000 7f950c9a4700 1 -- 172.21.15.93:0/1315381638 wait complete. 2021-11-20T08:56:33.608 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ls' 2021-11-20T08:56:33.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:33 smithi102 conmon[36472]: audit 2021-11-20T08:56:33.317156+0000 mgr.smithi093.qdikkm (mgr.14162) 170 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:33.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:33 smithi102 conmon[36472]: cluster 2021-11-20T08:56:33.519972+0000 mgr.smithi093.qdikkm (mgr.14162) 171 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:33.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:33 smithi093 conmon[33522]: audit 2021-11-20T08:56:33.317156+0000 mgr.smithi093.qdikkm (mgr.14162) 170 2021-11-20T08:56:33.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:33 smithi093 conmon[33522]: : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:33.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:33 smithi093 conmon[33522]: cluster 2021-11-20T08:56:33.519972+0000 mgr.smithi093.qdikkm (mgr.14162) 171 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:35.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3978173747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 msgr2=0x7fbfe40f5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:35.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3978173747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe40f5fd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fbfd4004660 tx=0x7fbfd4009b30).stop 2021-11-20T08:56:35.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3978173747 shutdown_connections 2021-11-20T08:56:35.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3978173747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfe40f6550 0x7fbfe40f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:35.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3978173747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe40f5fd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3978173747 >> 172.21.15.93:0/3978173747 conn(0x7fbfe40f04c0 msgr2=0x7fbfe40f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3978173747 shutdown_connections 2021-11-20T08:56:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3978173747 wait complete. 2021-11-20T08:56:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.074+0000 7fbfea5ef700 1 Processor -- start 2021-11-20T08:56:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfea5ef700 1 -- start start 2021-11-20T08:56:35.078 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfea5ef700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe4109d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:35.078 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfea5ef700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfe40f6550 0x7fbfe410c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:35.078 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfea5ef700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40f56a0 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.078 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfea5ef700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40f57e0 con 0x7fbfe40f6550 2021-11-20T08:56:35.079 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe4109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:35.079 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe4109d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42986/0 (socket says 172.21.15.93:42986) 2021-11-20T08:56:35.079 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe3fff700 1 -- 172.21.15.93:0/3924174857 learned_addr learned my addr 172.21.15.93:0/3924174857 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:35.079 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe37fe700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfe40f6550 0x7fbfe410c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:35.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe3fff700 1 -- 172.21.15.93:0/3924174857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfe40f6550 msgr2=0x7fbfe410c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:35.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe3fff700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfe40f6550 0x7fbfe410c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:35.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe3fff700 1 -- 172.21.15.93:0/3924174857 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd4005040 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe3fff700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe4109d60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbfd4009830 tx=0x7fbfd4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:35.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.075+0000 7fbfe0ff9700 1 -- 172.21.15.93:0/3924174857 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd4018070 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.076+0000 7fbfe0ff9700 1 -- 172.21.15.93:0/3924174857 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfd4007ba0 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.076+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe410c7b0 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.076+0000 7fbfe0ff9700 1 -- 172.21.15.93:0/3924174857 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd401dea0 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.076+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe410cd10 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.077+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4045ad0 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.077+0000 7fbfe0ff9700 1 -- 172.21.15.93:0/3924174857 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbfd4007d10 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.077+0000 7fbfe0ff9700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfcc05c2b0 0x7fbfcc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:35.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.077+0000 7fbfe0ff9700 1 -- 172.21.15.93:0/3924174857 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbfd40779c0 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.077+0000 7fbfe37fe700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfcc05c2b0 0x7fbfcc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:35.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.078+0000 7fbfe37fe700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfcc05c2b0 0x7fbfcc05e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fbfd80041d0 tx=0x7fbfd8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.081+0000 7fbfe0ff9700 1 -- 172.21.15.93:0/3924174857 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfd404a9c0 con 0x7fbfe40f5bb0 2021-11-20T08:56:35.252 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.253+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfe40fdf20 con 0x7fbfcc05c2b0 2021-11-20T08:56:35.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.257+0000 7fbfe0ff9700 1 -- 172.21.15.93:0/3924174857 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1116 (secure 0 0 0) 0x7fbfd8004700 con 0x7fbfcc05c2b0 2021-11-20T08:56:35.257 INFO:teuthology.orchestra.run.smithi093.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2021-11-20T08:56:35.258 INFO:teuthology.orchestra.run.smithi093.stdout:alertmanager 1/1 40s ago 3m count:1 2021-11-20T08:56:35.258 INFO:teuthology.orchestra.run.smithi093.stdout:crash 2/2 50s ago 4m * 2021-11-20T08:56:35.258 INFO:teuthology.orchestra.run.smithi093.stdout:grafana 1/1 40s ago 3m count:1 2021-11-20T08:56:35.258 INFO:teuthology.orchestra.run.smithi093.stdout:mgr 2/2 50s ago 4m count:2 2021-11-20T08:56:35.259 INFO:teuthology.orchestra.run.smithi093.stdout:mon 2/2 50s ago 3m smithi093:172.21.15.93=smithi093;smithi102:172.21.15.102=smithi102;count:2 2021-11-20T08:56:35.259 INFO:teuthology.orchestra.run.smithi093.stdout:node-exporter 2/2 50s ago 3m * 2021-11-20T08:56:35.259 INFO:teuthology.orchestra.run.smithi093.stdout:osd.unmanaged 6/6 50s ago - 2021-11-20T08:56:35.259 INFO:teuthology.orchestra.run.smithi093.stdout:prometheus 1/1 40s ago 3m count:1 2021-11-20T08:56:35.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfcc05c2b0 msgr2=0x7fbfcc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:35.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfcc05c2b0 0x7fbfcc05e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fbfd80041d0 tx=0x7fbfd8009040).stop 2021-11-20T08:56:35.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 msgr2=0x7fbfe4109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:35.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe4109d60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbfd4009830 tx=0x7fbfd4009b30).stop 2021-11-20T08:56:35.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 shutdown_connections 2021-11-20T08:56:35.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfcc05c2b0 0x7fbfcc05e770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:35.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe40f5bb0 0x7fbfe4109d60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:35.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 --2- 172.21.15.93:0/3924174857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfe40f6550 0x7fbfe410c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:35.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 >> 172.21.15.93:0/3924174857 conn(0x7fbfe40f04c0 msgr2=0x7fbfe40fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:35.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 shutdown_connections 2021-11-20T08:56:35.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:35.259+0000 7fbfea5ef700 1 -- 172.21.15.93:0/3924174857 wait complete. 2021-11-20T08:56:35.610 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch host ls' 2021-11-20T08:56:35.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:35 smithi093 conmon[33522]: audit 2021-11-20T08:56:35.255157+0000 mgr.smithi093.qdikkm 2021-11-20T08:56:35.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:35 smithi093 conmon[33522]: (mgr.14162) 172 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:35.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:35 smithi093 conmon[33522]: cluster 2021-11-20T08:56:35.520253+0000 mgr.smithi093.qdikkm (mgr.14162) 173 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:35.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:35 smithi102 conmon[36472]: audit 2021-11-20T08:56:35.255157+0000 mgr.smithi093.qdikkm (mgr.14162) 172 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:35.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:35 smithi102 conmon[36472]: cluster 2021-11-20T08:56:35.520253+0000 mgr.smithi093.qdikkm (mgr.14162) 173 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:37.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 -- 172.21.15.93:0/2787781641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 msgr2=0x7faccc0f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:37.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 --2- 172.21.15.93:0/2787781641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc0f6b40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7facb8004660 tx=0x7facb8009b30).stop 2021-11-20T08:56:37.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 -- 172.21.15.93:0/2787781641 shutdown_connections 2021-11-20T08:56:37.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 --2- 172.21.15.93:0/2787781641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc0f6b40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:37.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 --2- 172.21.15.93:0/2787781641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0f5440 0x7faccc0f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:37.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 -- 172.21.15.93:0/2787781641 >> 172.21.15.93:0/2787781641 conn(0x7faccc0f0ba0 msgr2=0x7faccc0f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:37.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 -- 172.21.15.93:0/2787781641 shutdown_connections 2021-11-20T08:56:37.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.014+0000 7facd1ef0700 1 -- 172.21.15.93:0/2787781641 wait complete. 2021-11-20T08:56:37.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7facd1ef0700 1 Processor -- start 2021-11-20T08:56:37.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7facd1ef0700 1 -- start start 2021-11-20T08:56:37.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7facd1ef0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0f5440 0x7faccc109ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:37.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7facd1ef0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc10bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:37.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7facd1ef0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc0603c0 con 0x7faccc0f66e0 2021-11-20T08:56:37.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7facd1ef0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc060500 con 0x7faccc0f5440 2021-11-20T08:56:37.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7faccbfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc10bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:37.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7faccbfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc10bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:42998/0 (socket says 172.21.15.93:42998) 2021-11-20T08:56:37.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7faccbfff700 1 -- 172.21.15.93:0/683757141 learned_addr learned my addr 172.21.15.93:0/683757141 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:37.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7facd0eee700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0f5440 0x7faccc109ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:37.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7faccbfff700 1 -- 172.21.15.93:0/683757141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0f5440 msgr2=0x7faccc109ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:37.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.015+0000 7faccbfff700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0f5440 0x7faccc109ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:37.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7faccbfff700 1 -- 172.21.15.93:0/683757141 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb8005040 con 0x7faccc0f66e0 2021-11-20T08:56:37.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7facd0eee700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0f5440 0x7faccc109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:56:37.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7faccbfff700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc10bfd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7facb8009830 tx=0x7facb8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:37.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7facc97fa700 1 -- 172.21.15.93:0/683757141 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facb8018070 con 0x7faccc0f66e0 2021-11-20T08:56:37.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7facc97fa700 1 -- 172.21.15.93:0/683757141 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facb8013410 con 0x7faccc0f66e0 2021-11-20T08:56:37.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc10c520 con 0x7faccc0f66e0 2021-11-20T08:56:37.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7facc97fa700 1 -- 172.21.15.93:0/683757141 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facb801c620 con 0x7faccc0f66e0 2021-11-20T08:56:37.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.016+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc10ca50 con 0x7faccc0f66e0 2021-11-20T08:56:37.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.017+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc045ad0 con 0x7faccc0f66e0 2021-11-20T08:56:37.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.017+0000 7facc97fa700 1 -- 172.21.15.93:0/683757141 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7facb801c8e0 con 0x7faccc0f66e0 2021-11-20T08:56:37.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.017+0000 7facc97fa700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7facbc05c300 0x7facbc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:37.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.018+0000 7facc97fa700 1 -- 172.21.15.93:0/683757141 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7facb8077a60 con 0x7faccc0f66e0 2021-11-20T08:56:37.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.018+0000 7facd0eee700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7facbc05c300 0x7facbc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:37.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.018+0000 7facd0eee700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7facbc05c300 0x7facbc05e7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7faccc060040 tx=0x7facc0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:37.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.021+0000 7facc97fa700 1 -- 172.21.15.93:0/683757141 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facb804aa00 con 0x7faccc0f66e0 2021-11-20T08:56:37.193 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.194+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7faccc0fdce0 con 0x7facbc05c300 2021-11-20T08:56:37.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.195+0000 7facc97fa700 1 -- 172.21.15.93:0/683757141 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+126 (secure 0 0 0) 0x7faccc0fdce0 con 0x7facbc05c300 2021-11-20T08:56:37.195 INFO:teuthology.orchestra.run.smithi093.stdout:HOST ADDR LABELS STATUS 2021-11-20T08:56:37.196 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 172.21.15.93 2021-11-20T08:56:37.196 INFO:teuthology.orchestra.run.smithi093.stdout:smithi102 smithi102 2021-11-20T08:56:37.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7facbc05c300 msgr2=0x7facbc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:37.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7facbc05c300 0x7facbc05e7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7faccc060040 tx=0x7facc0009040).stop 2021-11-20T08:56:37.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 msgr2=0x7faccc10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:37.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc10bfd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7facb8009830 tx=0x7facb8009b30).stop 2021-11-20T08:56:37.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 shutdown_connections 2021-11-20T08:56:37.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7facbc05c300 0x7facbc05e7c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:37.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0f5440 0x7faccc109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:37.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 --2- 172.21.15.93:0/683757141 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faccc0f66e0 0x7faccc10bfd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:37.200 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 >> 172.21.15.93:0/683757141 conn(0x7faccc0f0ba0 msgr2=0x7faccc0fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:37.200 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 shutdown_connections 2021-11-20T08:56:37.200 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:37.197+0000 7facd1ef0700 1 -- 172.21.15.93:0/683757141 wait complete. 2021-11-20T08:56:37.512 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch device ls' 2021-11-20T08:56:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:37 smithi093 conmon[33522]: audit 2021-11-20T08:56:37.196040+0000 mgr.smithi093.qdikkm 2021-11-20T08:56:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:37 smithi093 conmon[33522]: (mgr.14162) 174 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:37.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:37 smithi093 conmon[33522]: cluster 2021-11-20T08:56:37.520561+0000 mgr.smithi093.qdikkm (mgr.14162) 175 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:37.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:37 smithi102 conmon[36472]: audit 2021-11-20T08:56:37.196040+0000 mgr.smithi093.qdikkm (mgr.14162) 174 : audit [DBG] 2021-11-20T08:56:37.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:37 smithi102 conmon[36472]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:37.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:37 smithi102 conmon[36472]: cluster 2021-11-20T08:56:37.520561+0000 mgr.smithi093.qdikkm (mgr.14162) 175 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:38.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 -- 172.21.15.93:0/470624983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 msgr2=0x7fbb6c0f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:38.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 --2- 172.21.15.93:0/470624983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 0x7fbb6c0f6ae0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fbb68004660 tx=0x7fbb68009b30).stop 2021-11-20T08:56:38.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 -- 172.21.15.93:0/470624983 shutdown_connections 2021-11-20T08:56:38.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 --2- 172.21.15.93:0/470624983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 0x7fbb6c0f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:38.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 --2- 172.21.15.93:0/470624983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 0x7fbb6c0f6ae0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:38.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 -- 172.21.15.93:0/470624983 >> 172.21.15.93:0/470624983 conn(0x7fbb6c0f1e20 msgr2=0x7fbb6c0f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:38.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 -- 172.21.15.93:0/470624983 shutdown_connections 2021-11-20T08:56:38.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.893+0000 7fbb734e6700 1 -- 172.21.15.93:0/470624983 wait complete. 2021-11-20T08:56:38.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb734e6700 1 Processor -- start 2021-11-20T08:56:38.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb734e6700 1 -- start start 2021-11-20T08:56:38.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb734e6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 0x7fbb6c106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:38.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb734e6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 0x7fbb6c1074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:38.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb734e6700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c107a50 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.898 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb734e6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c107b90 con 0x7fbb6c0f7960 2021-11-20T08:56:38.898 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb70a81700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 0x7fbb6c1074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:38.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb70a81700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 0x7fbb6c1074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.93:60168/0 (socket says 172.21.15.93:60168) 2021-11-20T08:56:38.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb70a81700 1 -- 172.21.15.93:0/2249616761 learned_addr learned my addr 172.21.15.93:0/2249616761 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:38.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb71282700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 0x7fbb6c106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:38.901 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb71282700 1 -- 172.21.15.93:0/2249616761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 msgr2=0x7fbb6c1074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:38.902 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb71282700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 0x7fbb6c1074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:38.902 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb71282700 1 -- 172.21.15.93:0/2249616761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68005040 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.902 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.894+0000 7fbb70a81700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 0x7fbb6c1074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T08:56:38.902 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.895+0000 7fbb71282700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 0x7fbb6c106fb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbb68009830 tx=0x7fbb68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:38.903 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.895+0000 7fbb61ffb700 1 -- 172.21.15.93:0/2249616761 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb68018070 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.905 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.895+0000 7fbb61ffb700 1 -- 172.21.15.93:0/2249616761 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb68013410 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.905 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.895+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c1084d0 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.905 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.895+0000 7fbb61ffb700 1 -- 172.21.15.93:0/2249616761 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb680123c0 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.906 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.895+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c108a30 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.906 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.896+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c045ad0 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.907 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.896+0000 7fbb61ffb700 1 -- 172.21.15.93:0/2249616761 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbb6801d420 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.907 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.896+0000 7fbb61ffb700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbb5805c2b0 0x7fbb5805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:38.908 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.897+0000 7fbb61ffb700 1 -- 172.21.15.93:0/2249616761 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbb68077870 con 0x7fbb6c0f66c0 2021-11-20T08:56:38.908 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.897+0000 7fbb70a81700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbb5805c2b0 0x7fbb5805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:38.908 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.897+0000 7fbb70a81700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbb5805c2b0 0x7fbb5805e770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbb5c007fd0 tx=0x7fbb5c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:38.908 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:38.900+0000 7fbb61ffb700 1 -- 172.21.15.93:0/2249616761 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb6804a8c0 con 0x7fbb6c0f66c0 2021-11-20T08:56:39.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.075+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb6c0feec0 con 0x7fbb5805c2b0 2021-11-20T08:56:39.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.077+0000 7fbb61ffb700 1 -- 172.21.15.93:0/2249616761 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+279 (secure 0 0 0) 0x7fbb6c0feec0 con 0x7fbb5805c2b0 2021-11-20T08:56:39.076 INFO:teuthology.orchestra.run.smithi093.stdout:Hostname Path Type Serial Size Health Ident Fault Available 2021-11-20T08:56:39.077 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/nvme0n1 ssd PHFT62040150400BGN 400G Unknown N/A N/A No 2021-11-20T08:56:39.077 INFO:teuthology.orchestra.run.smithi093.stdout:smithi102 /dev/nvme0n1 ssd PHFT620400VK400BGN 400G Unknown N/A N/A No 2021-11-20T08:56:39.079 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbb5805c2b0 msgr2=0x7fbb5805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:39.079 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbb5805c2b0 0x7fbb5805e770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbb5c007fd0 tx=0x7fbb5c009040).stop 2021-11-20T08:56:39.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 msgr2=0x7fbb6c106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:39.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 0x7fbb6c106fb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbb68009830 tx=0x7fbb68009b30).stop 2021-11-20T08:56:39.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 shutdown_connections 2021-11-20T08:56:39.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbb5805c2b0 0x7fbb5805e770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:39.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb6c0f66c0 0x7fbb6c106fb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:39.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 --2- 172.21.15.93:0/2249616761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb6c0f7960 0x7fbb6c1074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:39.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.078+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 >> 172.21.15.93:0/2249616761 conn(0x7fbb6c0f1e20 msgr2=0x7fbb6c0fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:39.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.079+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 shutdown_connections 2021-11-20T08:56:39.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:39.079+0000 7fbb734e6700 1 -- 172.21.15.93:0/2249616761 wait complete. 2021-11-20T08:56:39.417 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T08:56:39.429 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T08:56:39.430 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs volume create cephfs --placement=4' 2021-11-20T08:56:39.865 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:39 smithi093 conmon[33522]: audit 2021-11-20T08:56:39.077254+0000 mgr.smithi093.qdikkm (mgr.14162) 176 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:39.865 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:39 smithi093 conmon[33522]: cluster 2021-11-20T08:56:39.520871+0000 mgr.smithi093.qdikkm (mgr.14162) 177 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:39.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:39 smithi102 conmon[36472]: audit 2021-11-20T08:56:39.077254+0000 mgr.smithi093.qdikkm (mgr.14162) 176 : audit 2021-11-20T08:56:40.251 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:39 smithi102 conmon[36472]: [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:40.251 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:39 smithi102 conmon[36472]: cluster 2021-11-20T08:56:39.520871+0000 mgr.smithi093.qdikkm (mgr.14162) 177 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:40.828 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 -- 172.21.15.93:0/2814352869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f5430 msgr2=0x7f01e80f5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:40.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 --2- 172.21.15.93:0/2814352869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f5430 0x7f01e80f5850 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f01d8004660 tx=0x7f01d8009b30).stop 2021-11-20T08:56:40.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 -- 172.21.15.93:0/2814352869 shutdown_connections 2021-11-20T08:56:40.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 --2- 172.21.15.93:0/2814352869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01e80f66d0 0x7f01e80f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:40.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 --2- 172.21.15.93:0/2814352869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f5430 0x7f01e80f5850 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:40.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 -- 172.21.15.93:0/2814352869 >> 172.21.15.93:0/2814352869 conn(0x7f01e80f0b90 msgr2=0x7f01e80f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:40.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 -- 172.21.15.93:0/2814352869 shutdown_connections 2021-11-20T08:56:40.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.826+0000 7f01e759e700 1 -- 172.21.15.93:0/2814352869 wait complete. 2021-11-20T08:56:40.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e759e700 1 Processor -- start 2021-11-20T08:56:40.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e759e700 1 -- start start 2021-11-20T08:56:40.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01e80f5430 0x7f01e8109ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:40.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e759e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f66d0 0x7f01e810bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:40.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e759e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e80603c0 con 0x7f01e80f66d0 2021-11-20T08:56:40.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e759e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e8060500 con 0x7f01e80f5430 2021-11-20T08:56:40.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e5d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f66d0 0x7f01e810bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:40.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e5d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f66d0 0x7f01e810bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43016/0 (socket says 172.21.15.93:43016) 2021-11-20T08:56:40.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e5d9b700 1 -- 172.21.15.93:0/1803994705 learned_addr learned my addr 172.21.15.93:0/1803994705 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:40.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e659c700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01e80f5430 0x7f01e8109ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:40.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e5d9b700 1 -- 172.21.15.93:0/1803994705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01e80f5430 msgr2=0x7f01e8109ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:40.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.827+0000 7f01e5d9b700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01e80f5430 0x7f01e8109ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:40.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.828+0000 7f01e5d9b700 1 -- 172.21.15.93:0/1803994705 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8005040 con 0x7f01e80f66d0 2021-11-20T08:56:40.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.828+0000 7f01e5d9b700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f66d0 0x7f01e810bfd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f01dc00c530 tx=0x7f01dc007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:40.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.828+0000 7f01d6ffd700 1 -- 172.21.15.93:0/1803994705 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01dc00f650 con 0x7f01e80f66d0 2021-11-20T08:56:40.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.828+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e810c580 con 0x7f01e80f66d0 2021-11-20T08:56:40.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.828+0000 7f01d6ffd700 1 -- 172.21.15.93:0/1803994705 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01dc00fc60 con 0x7f01e80f66d0 2021-11-20T08:56:40.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.828+0000 7f01d6ffd700 1 -- 172.21.15.93:0/1803994705 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01dc012c30 con 0x7f01e80f66d0 2021-11-20T08:56:40.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.828+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e810cb10 con 0x7f01e80f66d0 2021-11-20T08:56:40.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.829+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e8059100 con 0x7f01e80f66d0 2021-11-20T08:56:40.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.829+0000 7f01d6ffd700 1 -- 172.21.15.93:0/1803994705 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f01dc010070 con 0x7f01e80f66d0 2021-11-20T08:56:40.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.829+0000 7f01d6ffd700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f01d005c300 0x7f01d005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:40.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.829+0000 7f01d6ffd700 1 -- 172.21.15.93:0/1803994705 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f01dc075810 con 0x7f01e80f66d0 2021-11-20T08:56:40.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.829+0000 7f01e659c700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f01d005c300 0x7f01d005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:40.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.830+0000 7f01e659c700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f01d005c300 0x7f01d005e7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f01e8060040 tx=0x7f01d8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:40.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:40.833+0000 7f01d6ffd700 1 -- 172.21.15.93:0/1803994705 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01dc0452b0 con 0x7f01e80f66d0 2021-11-20T08:56:41.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:41.009+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 --> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f01e80fdd40 con 0x7f01d005c300 2021-11-20T08:56:41.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:41 smithi102 conmon[36472]: audit 2021-11-20T08:56:41.011838+0000 mon.smithi093 (mon.0) 447 : audit 2021-11-20T08:56:41.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:41 smithi102 conmon[36472]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T08:56:41.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:41 smithi093 conmon[33522]: audit 2021-11-20T08:56:41. 2021-11-20T08:56:41.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:41 smithi093 conmon[33522]: 011838+0000 mon.smithi093 (mon.0) 447 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T08:56:42.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.342+0000 7f01d6ffd700 1 -- 172.21.15.93:0/1803994705 <== mgr.14162 v2:172.21.15.93:6800/3953495879 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f01e80fdd40 con 0x7f01d005c300 2021-11-20T08:56:42.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f01d005c300 msgr2=0x7f01d005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:42.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f01d005c300 0x7f01d005e7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f01e8060040 tx=0x7f01d8009b30).stop 2021-11-20T08:56:42.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f66d0 msgr2=0x7f01e810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:42.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f66d0 0x7f01e810bfd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f01dc00c530 tx=0x7f01dc007e20).stop 2021-11-20T08:56:42.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 shutdown_connections 2021-11-20T08:56:42.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f01d005c300 0x7f01d005e7c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:42.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01e80f5430 0x7f01e8109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:42.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 --2- 172.21.15.93:0/1803994705 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01e80f66d0 0x7f01e810bfd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:42.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 >> 172.21.15.93:0/1803994705 conn(0x7f01e80f0b90 msgr2=0x7f01e80fa120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:42.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 shutdown_connections 2021-11-20T08:56:42.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:42.344+0000 7f01e759e700 1 -- 172.21.15.93:0/1803994705 wait complete. 2021-11-20T08:56:42.686 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: debug 2021-11-20T08:56:42.317+0000 7f0179373700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:56:42.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021-11-20T08:56:41.011253+0000 mgr.smithi093.qdikkm (mgr.14162) 178 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:42.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021 2021-11-20T08:56:42.687 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: -11-20T08:56:41.307727+0000 mon.smithi093 (mon.0) 448 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T08:56:42.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cluster 2021-11-20T08:56:41.307780+0000 mon.smithi093 (mon.0) 449 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-20T08:56:42.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021-11-20T08:56:41.309001+0000 mon.smithi093 ( 2021-11-20T08:56:42.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: mon.0) 450 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T08:56:42.688 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cluster 2021-11-20T08:56:42.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 2021-11-20T08:56:41.521181 2021-11-20T08:56:42.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 179 : cluster [DBG] 2021-11-20T08:56:42.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: pgmap v143: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:42.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021-11-20T08:56:42.689 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 2021-11-20T08: 2021-11-20T08:56:42.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 56:42.311826+0000 mon.smithi093 (mon.0) 2021-11-20T08:56:42.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 451 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T08:56:42.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:56:42.690 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 08:56:42.312024+0000 mon.smithi093 (mon.0 2021-11-20T08:56:42.691 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: ) 452 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-20T08:56:42.691 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:56:42.691 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 20T08:56:42.317452+0000 mon.smithi093 (mon 2021-11-20T08:56:42.691 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: .0) 453 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T08:56:42.692 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:56:42.692 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 11-20T08:56:42.318293+0000 2021-11-20T08:56:42.692 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: mon.smithi093 (mon.0) 454 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:56:42.692 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 2021-11-20T08:56:42.693 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cluster 2021-11-20T08:56:42.693 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 2021-11-20T08:56:42.318323+0000 mon.smithi093 ( 2021-11-20T08:56:42.693 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: mon.0) 455 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T08:56:42.693 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:56:42.694 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 11-20T08:56:42.327930+0000 mon.smithi093 2021-11-20T08:56:42.694 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: (mon.0) 456 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T08:56:42.694 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:56:42.694 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 11-20T08:56:42.328944+0000 mon.smithi093 2021-11-20T08:56:42.695 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: (mon.0) 457 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T08:56:42.695 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cluster 2021-11-20T08:56:42.695 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 2021-11-20T08:56:42.329038+0000 2021-11-20T08:56:42.695 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: mon.smithi093 (mon.0) 458 : cluster [DBG] 2021-11-20T08:56:42.695 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: fsmap cephfs:0 2021-11-20T08:56:42.696 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: cephadm 2021-11-20T08:56:42.696 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 2021-11-20T08:56:42.334929+0000 mgr.smithi093.qdikkm 2021-11-20T08:56:42.696 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: (mgr.14162) 180 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T08:56:42.696 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021-11-20T08:56:42.697 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 2021-11-20T08 2021-11-20T08:56:42.697 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: :56:42.342329+0000 mon.smithi093 (mon.0 2021-11-20T08:56:42.698 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: ) 459 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:42.698 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:56:42.698 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 20T08:56:42. 2021-11-20T08:56:42.698 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: 343940+0000 mon.smithi093 (mon.0) 460 : audit 2021-11-20T08:56:42.699 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:42 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:56:42.749 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T08:56:42.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021-11-20T08:56:41.011253+0000 mgr.smithi093.qdikkm (mgr.14162) 178 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T08:56:42.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021-11-20T08:56:41.307727+0000 mon.smithi093 (mon.0) 448 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T08:56:42.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cluster 2021-11-20T08 2021-11-20T08:56:42.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: :56:41.307780+0000 mon.smithi093 (mon.0) 449 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-20T08:56:42.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021-11-20T08:56:42.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 2021-11-20T08:56 2021-11-20T08:56:42.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: :41.309001+0000 mon.smithi093 (mon.0) 450 2021-11-20T08:56:42.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T08:56:42.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:56:42.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: -11-20T08:56:41. 2021-11-20T08:56:42.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 521181+0000 mgr.smithi093.qdikkm (mgr.14162) 179 : cluster 2021-11-20T08:56:42.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: [DBG] pgmap v143: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:42.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:56:42.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 20T08:56:42.311826+0000 2021-11-20T08:56:42.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: mon.smithi093 (mon.0) 451 : audit [INF] 2021-11-20T08:56:42.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T08:56:42.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cluster 2021-11-20T08:56:42.312024+0000 2021-11-20T08:56:42.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: mon.smithi093 (mon.0) 452 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-20T08:56:42.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021-11-20T08:56:42.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 2021-11-20T08:56: 2021-11-20T08:56:42.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 42.317452+0000 mon.smithi093 (mon.0) 453 : 2021-11-20T08:56:42.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T08:56:42.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:56:42.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: T08:56:42.318293+0000 mon.smithi093 2021-11-20T08:56:42.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: (mon.0) 454 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:56:42.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 2021-11-20T08:56:42.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cluster 2021-11-20T08 2021-11-20T08:56:42.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: :56:42.318323+0000 mon.smithi093 (mon. 2021-11-20T08:56:42.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 0) 455 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T08:56:42.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:56:42.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: -11-20T08:56: 2021-11-20T08:56:42.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 42.327930+0000 mon.smithi093 (mon.0) 456 : cluster 2021-11-20T08:56:42.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T08:56:42.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:56:42.928 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 20T08:56:42.328944+0000 2021-11-20T08:56:42.928 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: mon.smithi093 (mon.0) 457 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T08:56:42.928 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 2021-11-20T08:56:42.928 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T08:56:42.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 08:56:42.329038+0000 mon.smithi093 ( 2021-11-20T08:56:42.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: mon.0) 458 : cluster [DBG] fsmap cephfs:0 2021-11-20T08:56:42.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cephadm 2021-11-20T08:56:42.929 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 2021-11-20T08:56: 2021-11-20T08:56:42.930 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 42.334929+0000 mgr.smithi093.qdikkm (mgr.14162) 180 : 2021-11-20T08:56:42.930 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T08:56:42.930 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021 2021-11-20T08:56:42.930 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: -11-20T08:56: 2021-11-20T08:56:42.931 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: 42.342329+0000 mon.smithi093 (mon.0) 459 : audit 2021-11-20T08:56:42.931 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:42.931 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:56:42.931 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: T08:56:42.343940+0000 2021-11-20T08:56:42.932 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: mon.smithi093 (mon.0) 460 : audit [DBG] 2021-11-20T08:56:42.932 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:42 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:56:43.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: audit 2021-11-20T08:56:42.812530+0000 mon.smithi093 (mon.0) 461 2021-11-20T08:56:43.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:43.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: audit 2021-11-20T08:56:42.820445+0000 mon.smithi093 (mon.0) 462 2021-11-20T08:56:43.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:43.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: audit 2021-11-20T08:56:43.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 2021-11-20T08:56:42 2021-11-20T08:56:43.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: .821197+0000 mon.smithi093 (mon.0) 463 : audit 2021-11-20T08:56:43.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.lhrlmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:43.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:56:43.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: T08:56:42.826226+0000 2021-11-20T08:56:43.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: mon.smithi093 (mon.0) 464 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.lhrlmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:43.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 2021-11-20T08:56:43.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: audit 2021-11-20T08:56:43.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 2021-11-20T08:56: 2021-11-20T08:56:43.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 42.826990+0000 mon.smithi093 (mon.0) 465 : 2021-11-20T08:56:43.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:43.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: cephadm 2021-11-20 2021-11-20T08:56:43.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: T08:56:42.827698+0000 2021-11-20T08:56:43.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 181 : cephadm [INF] Deploying daemon mds.cephfs.smithi102.lhrlmh on smithi102 2021-11-20T08:56:43.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 2021-11-20T08:56:43.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: cluster 2021-11-20T08:56:43.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 2021-11-20T08:56:43. 2021-11-20T08:56:43.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 331701+0000 mon.smithi093 (mon.0) 466 : cluster 2021-11-20T08:56:43.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T08:56:43.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T08:56:43.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: 20T08:56:43.521654+0000 2021-11-20T08:56:43.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 182 : cluster [DBG] 2021-11-20T08:56:43.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:43 smithi102 conmon[36472]: pgmap v147: 65 pgs: 3 creating+activating, 12 creating+peering, 40 unknown, 10 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:44.107 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: audit 2021-11-20T08:56:42.812530+0000 mon.smithi093 (mon.0) 461 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:44.107 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: 2021-11-20T08:56:44.107 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: audit 2021-11-20T08:56:42.820445+0000 mon.smithi093 (mon.0) 462 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:44.108 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: audit 2021-11-20T08:56:42.821197+0000 mon.smithi093 (mon.0) 463 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.lhrlmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:44.108 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: audit 2021-11-20T08:56:42.826226+0000 mon.smithi093 (mon.0) 464 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.lhrlmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:44.108 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: audit 2021-11-20T08:56:42.826990+0000 mon.smithi093 (mon.0) 465 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:44.108 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: cephadm 2021-11-20T08:56:42.827698+0000 mgr.smithi093.qdikkm (mgr.14162) 181 : cephadm [INF] Deploying daemon mds.cephfs.smithi102.lhrlmh on smithi102 2021-11-20T08:56:44.109 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: cluster 2021-11-20T08:56:43.331701+0000 mon.smithi093 (mon.0) 466 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T08:56:44.109 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:43 smithi093 conmon[33522]: cluster 2021-11-20T08:56:43.521654+0000 mgr.smithi093.qdikkm (mgr.14162) 182 : cluster [DBG] pgmap v147: 65 pgs: 3 creating+activating, 12 creating+peering, 40 unknown, 10 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:44.243 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 -- 172.21.15.93:0/1298594290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 msgr2=0x7f78040f5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:44.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 --2- 172.21.15.93:0/1298594290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78040f5e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f77f8004660 tx=0x7f77f8009b30).stop 2021-11-20T08:56:44.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 -- 172.21.15.93:0/1298594290 shutdown_connections 2021-11-20T08:56:44.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 --2- 172.21.15.93:0/1298594290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78040f63b0 0x7f78040f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:44.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 --2- 172.21.15.93:0/1298594290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78040f5e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:44.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 -- 172.21.15.93:0/1298594290 >> 172.21.15.93:0/1298594290 conn(0x7f78040f0320 msgr2=0x7f78040f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:44.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 -- 172.21.15.93:0/1298594290 shutdown_connections 2021-11-20T08:56:44.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.240+0000 7f7809873700 1 -- 172.21.15.93:0/1298594290 wait complete. 2021-11-20T08:56:44.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7809873700 1 Processor -- start 2021-11-20T08:56:44.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7809873700 1 -- start start 2021-11-20T08:56:44.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7809873700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78041099f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:44.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7809873700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78040f63b0 0x7f780410bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:44.247 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7809873700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78040f5540 con 0x7f78040f5a10 2021-11-20T08:56:44.247 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7809873700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78040f5680 con 0x7f78040f63b0 2021-11-20T08:56:44.247 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7808871700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78041099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:44.247 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7808871700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78041099f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43024/0 (socket says 172.21.15.93:43024) 2021-11-20T08:56:44.248 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7808871700 1 -- 172.21.15.93:0/778352191 learned_addr learned my addr 172.21.15.93:0/778352191 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:44.248 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.241+0000 7f7803fff700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78040f63b0 0x7f780410bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:44.248 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f7808871700 1 -- 172.21.15.93:0/778352191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78040f63b0 msgr2=0x7f780410bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:44.248 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f7808871700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78040f63b0 0x7f780410bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:44.249 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f7808871700 1 -- 172.21.15.93:0/778352191 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f8005040 con 0x7f78040f5a10 2021-11-20T08:56:44.249 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f7808871700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78041099f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f77f8000c00 tx=0x7f77f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:44.249 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f78017fa700 1 -- 172.21.15.93:0/778352191 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f8019070 con 0x7f78040f5a10 2021-11-20T08:56:44.251 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f78017fa700 1 -- 172.21.15.93:0/778352191 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77f8007e70 con 0x7f78040f5a10 2021-11-20T08:56:44.251 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780410c4a0 con 0x7f78040f5a10 2021-11-20T08:56:44.251 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f78017fa700 1 -- 172.21.15.93:0/778352191 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f8014d40 con 0x7f78040f5a10 2021-11-20T08:56:44.251 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.242+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780410c9d0 con 0x7f78040f5a10 2021-11-20T08:56:44.252 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.243+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7804059100 con 0x7f78040f5a10 2021-11-20T08:56:44.252 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.243+0000 7f78017fa700 1 -- 172.21.15.93:0/778352191 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f77f8025070 con 0x7f78040f5a10 2021-11-20T08:56:44.253 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.244+0000 7f78017fa700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f77f405c300 0x7f77f405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:44.253 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.244+0000 7f78017fa700 1 -- 172.21.15.93:0/778352191 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f77f8076d80 con 0x7f78040f5a10 2021-11-20T08:56:44.253 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.244+0000 7f7803fff700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f77f405c300 0x7f77f405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:44.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.244+0000 7f7803fff700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f77f405c300 0x7f77f405e7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f77f0009fd0 tx=0x7f77f000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:44.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.248+0000 7f78017fa700 1 -- 172.21.15.93:0/778352191 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77f8049ac0 con 0x7f78040f5a10 2021-11-20T08:56:44.438 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.439+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f78040fdce0 con 0x7f78040f5a10 2021-11-20T08:56:44.438 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.439+0000 7f78017fa700 1 -- 172.21.15.93:0/778352191 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1003 (secure 0 0 0) 0x7f77f8078070 con 0x7f78040f5a10 2021-11-20T08:56:44.439 INFO:teuthology.orchestra.run.smithi093.stdout:e2 2021-11-20T08:56:44.439 INFO:teuthology.orchestra.run.smithi093.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T08:56:44.440 INFO:teuthology.orchestra.run.smithi093.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T08:56:44.440 INFO:teuthology.orchestra.run.smithi093.stdout:legacy client fscid: 1 2021-11-20T08:56:44.440 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:44.440 INFO:teuthology.orchestra.run.smithi093.stdout:Filesystem 'cephfs' (1) 2021-11-20T08:56:44.441 INFO:teuthology.orchestra.run.smithi093.stdout:fs_name cephfs 2021-11-20T08:56:44.441 INFO:teuthology.orchestra.run.smithi093.stdout:epoch 2 2021-11-20T08:56:44.441 INFO:teuthology.orchestra.run.smithi093.stdout:flags 12 2021-11-20T08:56:44.441 INFO:teuthology.orchestra.run.smithi093.stdout:created 2021-11-20T08:56:42.318132+0000 2021-11-20T08:56:44.442 INFO:teuthology.orchestra.run.smithi093.stdout:modified 2021-11-20T08:56:42.318202+0000 2021-11-20T08:56:44.442 INFO:teuthology.orchestra.run.smithi093.stdout:tableserver 0 2021-11-20T08:56:44.442 INFO:teuthology.orchestra.run.smithi093.stdout:root 0 2021-11-20T08:56:44.442 INFO:teuthology.orchestra.run.smithi093.stdout:session_timeout 60 2021-11-20T08:56:44.442 INFO:teuthology.orchestra.run.smithi093.stdout:session_autoclose 300 2021-11-20T08:56:44.443 INFO:teuthology.orchestra.run.smithi093.stdout:max_file_size 1099511627776 2021-11-20T08:56:44.443 INFO:teuthology.orchestra.run.smithi093.stdout:required_client_features {} 2021-11-20T08:56:44.443 INFO:teuthology.orchestra.run.smithi093.stdout:last_failure 0 2021-11-20T08:56:44.443 INFO:teuthology.orchestra.run.smithi093.stdout:last_failure_osd_epoch 0 2021-11-20T08:56:44.444 INFO:teuthology.orchestra.run.smithi093.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T08:56:44.444 INFO:teuthology.orchestra.run.smithi093.stdout:max_mds 1 2021-11-20T08:56:44.444 INFO:teuthology.orchestra.run.smithi093.stdout:in 2021-11-20T08:56:44.444 INFO:teuthology.orchestra.run.smithi093.stdout:up {} 2021-11-20T08:56:44.444 INFO:teuthology.orchestra.run.smithi093.stdout:failed 2021-11-20T08:56:44.445 INFO:teuthology.orchestra.run.smithi093.stdout:damaged 2021-11-20T08:56:44.445 INFO:teuthology.orchestra.run.smithi093.stdout:stopped 2021-11-20T08:56:44.445 INFO:teuthology.orchestra.run.smithi093.stdout:data_pools [3] 2021-11-20T08:56:44.445 INFO:teuthology.orchestra.run.smithi093.stdout:metadata_pool 2 2021-11-20T08:56:44.446 INFO:teuthology.orchestra.run.smithi093.stdout:inline_data disabled 2021-11-20T08:56:44.446 INFO:teuthology.orchestra.run.smithi093.stdout:balancer 2021-11-20T08:56:44.446 INFO:teuthology.orchestra.run.smithi093.stdout:standby_count_wanted 0 2021-11-20T08:56:44.446 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:44.447 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:44.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f77f405c300 msgr2=0x7f77f405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:44.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f77f405c300 0x7f77f405e7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f77f0009fd0 tx=0x7f77f000b040).stop 2021-11-20T08:56:44.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 msgr2=0x7f78041099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:44.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78041099f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f77f8000c00 tx=0x7f77f800b040).stop 2021-11-20T08:56:44.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 shutdown_connections 2021-11-20T08:56:44.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f77f405c300 0x7f77f405e7c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:44.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78040f5a10 0x7f78041099f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:44.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 --2- 172.21.15.93:0/778352191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78040f63b0 0x7f780410bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:44.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 >> 172.21.15.93:0/778352191 conn(0x7f78040f0320 msgr2=0x7f78040fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:44.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 shutdown_connections 2021-11-20T08:56:44.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:44.442+0000 7f7809873700 1 -- 172.21.15.93:0/778352191 wait complete. 2021-11-20T08:56:44.451 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 2 2021-11-20T08:56:44.866 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T08:56:44.879 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T08:56:44.879 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs max_mds 1' 2021-11-20T08:56:45.594 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:45 smithi093 conmon[33522]: cluster 2021-11-20T08:56:44.332748+0000 mon.smithi093 2021-11-20T08:56:45.595 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:45 smithi093 conmon[33522]: (mon.0) 467 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-20T08:56:45.595 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:45 smithi093 conmon[33522]: audit 2021-11-20T08:56:44.440385+0000 mon.smithi093 (mon.0) 468 : audit [DBG] from='client.? 172.21.15.93:0/778352191' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T08:56:45.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:45 smithi093 conmon[33522]: audit 2021-11-20T08:56:44.894878+0000 mon.smithi093 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:45.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:45 smithi093 conmon[33522]: audit 2021-11-20T08:56:44.895639+0000 mon.smithi093 (mon.0) 470 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.ljzugc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:45.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:45 smithi093 conmon[33522]: audit 2021-11-20T08:56:44.900374+0000 mon.smithi093 (mon.0) 471 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.ljzugc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:45.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:45 smithi093 conmon[33522]: audit 2021-11-20T08:56:44.901123+0000 mon.smithi093 (mon.0) 472 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:45.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: cluster 2021-11-20T08:56:44.332748+0000 mon.smithi093 (mon.0) 2021-11-20T08:56:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: 467 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-20T08:56:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: audit 2021-11-20T08:56:44.440385+0000 mon.smithi093 (mon.0) 468 : audit [DBG] from='client.? 172.21.15.93:0/778352191' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T08:56:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T08:56:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: 08:56:44.894878+0000 mon.smithi093 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: audit 2021-11-20T08: 2021-11-20T08:56:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: 56:44.895639+0000 mon.smithi093 (mon.0) 470 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.ljzugc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: audit 2021-11-20T08:56:44.900374+0000 mon.smithi093 (mon.0 2021-11-20T08:56:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: ) 471 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.ljzugc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:45.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: audit 2021-11-20T08:56:44.901123+0000 mon.smithi093 (mon. 2021-11-20T08:56:45.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:45 smithi102 conmon[36472]: 0) 472 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:46.420 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cephadm 2021-11-20T08:56:44.901773+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:56:46.420 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: .14162) 183 : cephadm [INF] Deploying daemon mds.cephfs.smithi093.ljzugc on smithi093 2021-11-20T08:56:46.421 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.338111+0000 mon.smithi093 (mon.0) 473 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] up:boot 2021-11-20T08:56:46.421 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.338167+0000 mon.smithi093 (mon.0) 474 : cluster [INF] daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T08:56:46.421 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.338227+0000 mon.smithi093 (mon.0) 475 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:56:46.421 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.338248+0000 mon.smithi093 (mon.0) 476 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T08:56:46.422 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.338277+0000 mon.smithi093 (mon.0) 477 : cluster [INF] Cluster is now healthy 2021-11-20T08:56:46.422 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.338455+0000 mon.smithi093 (mon.0) 478 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T08:56:46.422 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: audit 2021-11-20T08:56:45.338539+0000 mon.smithi093 (mon.0) 479 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T08:56:46.422 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.342315+0000 mon.smithi093 (mon.0) 480 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:creating} 2021-11-20T08:56:46.423 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.356470+0000 mon.smithi093 (mon.0) 481 : cluster [INF] daemon mds.cephfs.smithi102.lhrlmh is now active in filesystem cephfs as rank 0 2021-11-20T08:56:46.423 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:46 smithi093 conmon[33522]: cluster 2021-11-20T08:56:45.522213+0000 mgr.smithi093.qdikkm (mgr.14162) 184 : cluster [DBG] pgmap v149: 65 pgs: 3 creating+activating, 12 creating+peering, 9 unknown, 41 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:46.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cephadm 2021-11-20T08:56:44.901773+0000 mgr.smithi093.qdikkm (mgr.14162) 183 : cephadm [INF] 2021-11-20T08:56:46.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: Deploying daemon mds.cephfs.smithi093.ljzugc on smithi093 2021-11-20T08:56:46.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:45.338111+0000 mon.smithi093 (mon.0) 473 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] up:boot 2021-11-20T08:56:46.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:45.338167+0000 mon.smithi093 (mon.0) 474 : 2021-11-20T08:56:46.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster [INF] daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T08:56:46.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:45.338227+0000 mon.smithi093 (mon.0) 475 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:56:46.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: 2021-11-20T08:56:46.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:45.338248+0000 mon.smithi093 (mon. 2021-11-20T08:56:46.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: 0) 476 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T08:56:46.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T08:56:46.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: 08:56:45.338277+0000 mon.smithi093 (mon.0) 477 : cluster [INF] Cluster is now healthy 2021-11-20T08:56:46.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:45.338455+0000 mon.smithi093 (mon. 2021-11-20T08:56:46.674 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: 0) 478 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T08:56:46.675 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: audit 2021-11-20T08:56:46.675 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: 2021-11-20T08:56:45.338539+0000 mon.smithi093 (mon.0) 479 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T08:56:46.675 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:45.342315 2021-11-20T08:56:46.676 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 480 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:creating} 2021-11-20T08:56:46.676 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:45.356470+0000 mon.smithi093 (mon.0) 2021-11-20T08:56:46.676 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: 481 : cluster [INF] daemon mds.cephfs.smithi102.lhrlmh is now active in filesystem cephfs as rank 0 2021-11-20T08:56:46.677 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: cluster 2021-11-20T08:56:46.678 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:46 smithi102 conmon[36472]: 2021-11-20T08:56:45.522213+0000 mgr.smithi093.qdikkm (mgr.14162) 184 : cluster [DBG] pgmap v149: 65 pgs: 3 creating+activating, 12 creating+peering, 9 unknown, 41 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:56:46.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 -- 172.21.15.93:0/3090465515 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f49a0 msgr2=0x7fefb40f4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:46.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 --2- 172.21.15.93:0/3090465515 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f49a0 0x7fefb40f4dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fefa4004660 tx=0x7fefa4009b30).stop 2021-11-20T08:56:46.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 -- 172.21.15.93:0/3090465515 shutdown_connections 2021-11-20T08:56:46.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 --2- 172.21.15.93:0/3090465515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefb40f5340 0x7fefb40f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:46.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 --2- 172.21.15.93:0/3090465515 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f49a0 0x7fefb40f4dc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:46.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 -- 172.21.15.93:0/3090465515 >> 172.21.15.93:0/3090465515 conn(0x7fefb40f04c0 msgr2=0x7fefb40f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:46.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 -- 172.21.15.93:0/3090465515 shutdown_connections 2021-11-20T08:56:46.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.833+0000 7fefb9015700 1 -- 172.21.15.93:0/3090465515 wait complete. 2021-11-20T08:56:46.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb9015700 1 Processor -- start 2021-11-20T08:56:46.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb9015700 1 -- start start 2021-11-20T08:56:46.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb9015700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefb40f49a0 0x7fefb40f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:46.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb9015700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f5340 0x7fefb40f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:46.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb9015700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40f5f30 con 0x7fefb40f5340 2021-11-20T08:56:46.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb9015700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40f6070 con 0x7fefb40f49a0 2021-11-20T08:56:46.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f5340 0x7fefb40f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:46.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f5340 0x7fefb40f59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43036/0 (socket says 172.21.15.93:43036) 2021-11-20T08:56:46.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb259c700 1 -- 172.21.15.93:0/4042757692 learned_addr learned my addr 172.21.15.93:0/4042757692 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:46.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb259c700 1 -- 172.21.15.93:0/4042757692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefb40f49a0 msgr2=0x7fefb40f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:46.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb259c700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefb40f49a0 0x7fefb40f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:46.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.834+0000 7fefb259c700 1 -- 172.21.15.93:0/4042757692 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa4005040 con 0x7fefb40f5340 2021-11-20T08:56:46.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.835+0000 7fefb259c700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f5340 0x7fefb40f59d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fefa800c510 tx=0x7fefa8007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:46.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.835+0000 7fef9b7fe700 1 -- 172.21.15.93:0/4042757692 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefa800f650 con 0x7fefb40f5340 2021-11-20T08:56:46.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.835+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb40f6350 con 0x7fefb40f5340 2021-11-20T08:56:46.841 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.835+0000 7fef9b7fe700 1 -- 172.21.15.93:0/4042757692 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefa800fc60 con 0x7fefb40f5340 2021-11-20T08:56:46.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.835+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb410f000 con 0x7fefb40f5340 2021-11-20T08:56:46.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.835+0000 7fef9b7fe700 1 -- 172.21.15.93:0/4042757692 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefa8012c80 con 0x7fefb40f5340 2021-11-20T08:56:46.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.836+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb4045ad0 con 0x7fefb40f5340 2021-11-20T08:56:46.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.836+0000 7fef9b7fe700 1 -- 172.21.15.93:0/4042757692 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fefa8010070 con 0x7fefb40f5340 2021-11-20T08:56:46.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.837+0000 7fef9b7fe700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fef9c05c2b0 0x7fef9c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:46.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.837+0000 7fef9b7fe700 1 -- 172.21.15.93:0/4042757692 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fefa8075b60 con 0x7fefb40f5340 2021-11-20T08:56:46.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.837+0000 7fefb2d9d700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fef9c05c2b0 0x7fef9c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:46.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.837+0000 7fefb2d9d700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fef9c05c2b0 0x7fef9c05e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fefa4009f50 tx=0x7fefa4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:46.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:46.841+0000 7fef9b7fe700 1 -- 172.21.15.93:0/4042757692 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefa80452b0 con 0x7fefb40f5340 2021-11-20T08:56:47.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.045+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fefb40feeb0 con 0x7fefb40f5340 2021-11-20T08:56:47.556 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: cluster 2021-11-20T08:56:46.346384+0000 mon.smithi093 (mon.0) 2021-11-20T08:56:47.557 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: 482 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] up:active 2021-11-20T08:56:47.557 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: cluster 2021-11-20T08:56:46.346426+0000 mon.smithi093 (mon.0) 483 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 2021-11-20T08:56:47.558 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: audit 2021-11-20 2021-11-20T08:56:47.558 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: T08:56:47.047081+0000 mon.smithi093 (mon.0) 484 : audit [INF] from='client.? 172.21.15.93:0/4042757692' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2021-11-20T08:56:47.559 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:56:47.559 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: 11-20T08:56:47.310724+0000 mon.smithi093 2021-11-20T08:56:47.559 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: (mon.0) 485 : audit [INF] 2021-11-20T08:56:47.560 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:47.560 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: audit 2021-11-20T08:56:47.561 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: 2021-11-20T 2021-11-20T08:56:47.561 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: 08:56:47.311445+0000 mon.smithi093 (mon. 2021-11-20T08:56:47.562 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: 0) 486 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hjodmi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:47.562 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:56:47.563 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: 11-20T08:56:47.314112+0000 mon.smithi093 (mon.0) 487 : audit 2021-11-20T08:56:47.563 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hjodmi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:47.564 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: audit 2021-11-20T08:56:47 2021-11-20T08:56:47.564 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: .314630+0000 mon.smithi093 (mon.0) 488 : audit 2021-11-20T08:56:47.564 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:47 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:47.567 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.556+0000 7fef9b7fe700 1 -- 172.21.15.93:0/4042757692 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fefa80488a0 con 0x7fefb40f5340 2021-11-20T08:56:47.567 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.558+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fef9c05c2b0 msgr2=0x7fef9c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:47.568 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.558+0000 7fefb9015700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fef9c05c2b0 0x7fef9c05e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fefa4009f50 tx=0x7fefa4009b30).stop 2021-11-20T08:56:47.568 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.558+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f5340 msgr2=0x7fefb40f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:47.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.558+0000 7fefb9015700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f5340 0x7fefb40f59d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fefa800c510 tx=0x7fefa8007df0).stop 2021-11-20T08:56:47.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.559+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 shutdown_connections 2021-11-20T08:56:47.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.559+0000 7fefb9015700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fef9c05c2b0 0x7fef9c05e770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:47.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.559+0000 7fefb9015700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefb40f49a0 0x7fefb40f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:56:47.571 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.559+0000 7fefb9015700 1 --2- 172.21.15.93:0/4042757692 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb40f5340 0x7fefb40f59d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:47.571 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.559+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 >> 172.21.15.93:0/4042757692 conn(0x7fefb40f04c0 msgr2=0x7fefb40fb510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:47.571 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.559+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 shutdown_connections 2021-11-20T08:56:47.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:47.559+0000 7fefb9015700 1 -- 172.21.15.93:0/4042757692 wait complete. 2021-11-20T08:56:47.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: cluster 2021-11-20T08:56:46.346384+0000 mon.smithi093 (mon.0) 482 : cluster 2021-11-20T08:56:47.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: [DBG] mds.? [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] up:active 2021-11-20T08:56:47.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: cluster 2021-11-20T08:56:46.346426+0000 mon.smithi093 (mon.0) 483 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 2021-11-20T08:56:47.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: audit 2021-11-20T08:56:47.047081+0000 mon.smithi093 (mon.0) 484 : audit [INF] from='client.? 172.21.15.93:0/4042757692' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2021-11-20T08:56:47.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: audit 2021-11-20T08:56:47.310724+0000 mon.smithi093 (mon.0) 485 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:47.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: audit 2021-11-20T08:56:47.311445+0000 mon.smithi093 (mon.0) 486 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hjodmi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:47.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: audit 2021-11-20T08:56:47.314112+0000 mon.smithi093 (mon. 2021-11-20T08:56:47.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: 0) 487 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi102.hjodmi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:47.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:47 smithi102 conmon[36472]: audit 2021-11-20T08:56:47.314630+0000 mon.smithi093 (mon.0) 488 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:48.042 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T08:56:48.060 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T08:56:48.061 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2021-11-20T08:56:48.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: cephadm 2021-11-20T08:56:47.315107+0000 2021-11-20T08:56:48.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 185 : cephadm [INF] Deploying daemon mds.cephfs.smithi102.hjodmi on smithi102 2021-11-20T08:56:48.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: cluster 2021-11-20T08:56:47.522620+0000 mgr.smithi093.qdikkm (mgr.14162) 186 : cluster [DBG] pgmap v150: 65 pgs: 3 creating+activating, 62 active+clean; 1.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s wr, 8 op/s 2021-11-20T08:56:48.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: audit 2021-11-20T08:56:47.557005+0000 mon.smithi093 (mon.0) 489 : audit [INF] from='client.? 172.21.15.93:0/4042757692' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2021-11-20T08:56:48.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: cluster 2021-11-20T08:56:47.557057+0000 mon.smithi093 (mon.0) 490 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:boot 2021-11-20T08:56:48.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: cluster 2021-11-20T08:56:47.557367+0000 mon.smithi093 (mon.0) 491 : cluster [DBG] 2021-11-20T08:56:48.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 1 up:standby 2021-11-20T08:56:48.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: audit 2021-11-20T08:56:47.557507+0000 mon.smithi093 (mon.0) 492 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:56:48.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:56:48.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: 56:47.560823+0000 mon.smithi093 (mon.0) 493 2021-11-20T08:56:48.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:48 smithi102 conmon[36472]: : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 1 up:standby 2021-11-20T08:56:48.610 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: cephadm 2021-11-20T08:56:47.315107+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:56:48.611 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: 185 : cephadm [INF] Deploying daemon mds.cephfs.smithi102.hjodmi on smithi102 2021-11-20T08:56:48.611 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: cluster 2021-11-20T08:56:47.522620+0000 mgr.smithi093.qdikkm (mgr.14162) 186 : cluster [DBG] pgmap v150: 65 pgs: 3 creating+activating, 62 active+clean; 1.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s wr, 8 op/s 2021-11-20T08:56:48.611 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: audit 2021-11-20T08:56:47.557005+0000 mon.smithi093 (mon.0) 489 : audit [INF] from='client.? 172.21.15.93:0/4042757692' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2021-11-20T08:56:48.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: cluster 2021-11-20T08:56:47.557057+0000 mon.smithi093 (mon.0) 490 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:boot 2021-11-20T08:56:48.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: cluster 2021-11-20T08:56:47.557367+0000 mon.smithi093 (mon.0) 491 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 1 up:standby 2021-11-20T08:56:48.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: audit 2021-11-20T08:56:47.557507+0000 mon.smithi093 (mon.0) 492 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:56:48.612 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:48 smithi093 conmon[33522]: cluster 2021-11-20T08:56:47.560823+0000 mon.smithi093 (mon.0) 493 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 1 up:standby 2021-11-20T08:56:49.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.613+0000 7f635d015700 1 -- 172.21.15.93:0/2062060596 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 msgr2=0x7f63580f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:49.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.613+0000 7f635d015700 1 --2- 172.21.15.93:0/2062060596 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f59e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f6348004660 tx=0x7f6348009b30).stop 2021-11-20T08:56:49.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 -- 172.21.15.93:0/2062060596 shutdown_connections 2021-11-20T08:56:49.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 --2- 172.21.15.93:0/2062060596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63580f6860 0x7f63580f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:49.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 --2- 172.21.15.93:0/2062060596 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f59e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:49.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 -- 172.21.15.93:0/2062060596 >> 172.21.15.93:0/2062060596 conn(0x7f63580f0d40 msgr2=0x7f63580f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:49.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 -- 172.21.15.93:0/2062060596 shutdown_connections 2021-11-20T08:56:49.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 -- 172.21.15.93:0/2062060596 wait complete. 2021-11-20T08:56:49.617 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 Processor -- start 2021-11-20T08:56:49.617 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 -- start start 2021-11-20T08:56:49.617 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.614+0000 7f635d015700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:49.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f635d015700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63580f6860 0x7f63580f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:49.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f635d015700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63580f8100 con 0x7f63580f55c0 2021-11-20T08:56:49.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f635d015700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63580f8240 con 0x7f63580f6860 2021-11-20T08:56:49.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f6356d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:49.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f6356d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43054/0 (socket says 172.21.15.93:43054) 2021-11-20T08:56:49.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f6356d9d700 1 -- 172.21.15.93:0/1534599771 learned_addr learned my addr 172.21.15.93:0/1534599771 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:49.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f635659c700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63580f6860 0x7f63580f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:49.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f6356d9d700 1 -- 172.21.15.93:0/1534599771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63580f6860 msgr2=0x7f63580f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:49.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f6356d9d700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63580f6860 0x7f63580f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:49.620 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f6356d9d700 1 -- 172.21.15.93:0/1534599771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348005040 con 0x7f63580f55c0 2021-11-20T08:56:49.620 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f635659c700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63580f6860 0x7f63580f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T08:56:49.620 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f6356d9d700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f9720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6348004e30 tx=0x7f6348009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:49.620 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f633f7fe700 1 -- 172.21.15.93:0/1534599771 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6348018070 con 0x7f63580f55c0 2021-11-20T08:56:49.621 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f633f7fe700 1 -- 172.21.15.93:0/1534599771 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6348013410 con 0x7f63580f55c0 2021-11-20T08:56:49.621 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63580f84c0 con 0x7f63580f55c0 2021-11-20T08:56:49.621 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63580f8a20 con 0x7f63580f55c0 2021-11-20T08:56:49.621 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.615+0000 7f633f7fe700 1 -- 172.21.15.93:0/1534599771 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634801de20 con 0x7f63580f55c0 2021-11-20T08:56:49.622 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.616+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6358045ad0 con 0x7f63580f55c0 2021-11-20T08:56:49.622 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.617+0000 7f633f7fe700 1 -- 172.21.15.93:0/1534599771 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f634801d420 con 0x7f63580f55c0 2021-11-20T08:56:49.622 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.617+0000 7f633f7fe700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f634005c2b0 0x7f634005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:49.622 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.617+0000 7f633f7fe700 1 -- 172.21.15.93:0/1534599771 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f6348077b20 con 0x7f63580f55c0 2021-11-20T08:56:49.622 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.617+0000 7f635659c700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f634005c2b0 0x7f634005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:49.623 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.617+0000 7f635659c700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f634005c2b0 0x7f634005e770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f634c0103c0 tx=0x7f634c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:49.623 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.621+0000 7f633f7fe700 1 -- 172.21.15.93:0/1534599771 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f634804a7e0 con 0x7f63580f55c0 2021-11-20T08:56:49.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:49.832+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f63580fdd70 con 0x7f63580f55c0 2021-11-20T08:56:50.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.489+0000 7f633f7fe700 1 -- 172.21.15.93:0/1534599771 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f634804de00 con 0x7f63580f55c0 2021-11-20T08:56:50.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f634005c2b0 msgr2=0x7f634005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:50.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f634005c2b0 0x7f634005e770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f634c0103c0 tx=0x7f634c00b040).stop 2021-11-20T08:56:50.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 msgr2=0x7f63580f9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:50.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f9720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6348004e30 tx=0x7f6348009b30).stop 2021-11-20T08:56:50.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 shutdown_connections 2021-11-20T08:56:50.491 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f634005c2b0 0x7f634005e770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:50.491 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63580f55c0 0x7f63580f9720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:50.492 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 --2- 172.21.15.93:0/1534599771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63580f6860 0x7f63580f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:50.492 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 >> 172.21.15.93:0/1534599771 conn(0x7f63580f0d40 msgr2=0x7f635805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:50.492 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 shutdown_connections 2021-11-20T08:56:50.492 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:50.491+0000 7f635d015700 1 -- 172.21.15.93:0/1534599771 wait complete. 2021-11-20T08:56:50.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: cluster 2021-11-20T08:56:49.452940+0000 mon.smithi093 (mon.0) 494 2021-11-20T08:56:50.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:boot 2021-11-20T08:56:50.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: cluster 2021-11-20T08:56:49.453009+0000 mon.smithi093 (mon.0) 495 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 2 up:standby 2021-11-20T08:56:50.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: audit 2021 2021-11-20T08:56:50.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: -11-20T08:56:49.453134+0000 mon.smithi093 (mon.0) 496 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T08:56:50.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: audit 2021-11-20T08:56:50.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 2021-11-20T08:56:49.483020+0000 mon.smithi093 (mon.0) 2021-11-20T08:56:50.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 497 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:50.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:56:50.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 20T08:56:49.483660+0000 mon.smithi093 (mon. 2021-11-20T08:56:50.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 0) 498 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.zrryyo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:50.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:56:50.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: T08:56:49.486166+0000 mon.smithi093 (mon. 2021-11-20T08:56:50.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 0) 499 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.zrryyo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:50.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:56:50.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 11-20T08:56:49.486724+0000 2021-11-20T08:56:50.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: mon.smithi093 (mon.0) 500 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:50.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 2021-11-20T08:56:50.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: cephadm 2021-11-20T08:56 2021-11-20T08:56:50.674 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: :49.487346+0000 mgr.smithi093.qdikkm (mgr.14162) 187 2021-11-20T08:56:50.674 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: : cephadm [INF] Deploying daemon mds.cephfs.smithi093.zrryyo on smithi093 2021-11-20T08:56:50.675 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:56:50.675 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: T08:56:49.523047+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:56:50.675 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: ) 188 : cluster [DBG] pgmap v151: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s wr, 6 op/s 2021-11-20T08:56:50.675 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T08:56:50.676 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 08:56:49.833887+0000 mon.smithi093 (mon.0) 2021-11-20T08:56:50.676 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:50 smithi102 conmon[36472]: 501 : audit [INF] from='client.? 172.21.15.93:0/1534599771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2021-11-20T08:56:50.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: cluster 2021-11-20T08:56:49.452940+0000 2021-11-20T08:56:50.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: mon.smithi093 (mon.0) 494 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:boot 2021-11-20T08:56:50.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: cluster 2021-11-20T08:56:49.453009+0000 mon.smithi093 (mon.0) 495 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 2 up:standby 2021-11-20T08:56:50.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit 2021-11-20T08:56:49.453134+0000 mon.smithi093 (mon.0) 496 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T08:56:50.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit 2021-11-20T08:56:49.483020+0000 mon.smithi093 (mon.0) 497 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:50.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit 2021-11-20T08:56:49.483660+0000 mon.smithi093 (mon.0) 498 : audit 2021-11-20T08:56:50.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.zrryyo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T08:56:50.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit 2021-11-20T08:56:49.486166+0000 mon.smithi093 (mon.0) 499 : 2021-11-20T08:56:50.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi093.zrryyo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T08:56:50.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit 2021-11-20T08:56:50.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: 2021-11-20T08:56:49.486724+0000 mon.smithi093 (mon.0 2021-11-20T08:56:50.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: ) 500 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T08:56:50.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: cephadm 2021- 2021-11-20T08:56:50.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: 11-20T08:56:49.487346+0000 mgr.smithi093.qdikkm 2021-11-20T08:56:50.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: (mgr.14162) 187 : cephadm [INF] Deploying daemon mds.cephfs.smithi093.zrryyo on smithi093 2021-11-20T08:56:50.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: cluster 2021-11-20T08:56:50.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: 2021-11-20T08:56:49.523047+0000 2021-11-20T08:56:50.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 188 : cluster [DBG] 2021-11-20T08:56:50.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: pgmap v151: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s wr, 6 op/s 2021-11-20T08:56:50.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit 2021-11-20T08:56: 2021-11-20T08:56:50.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: 49.833887+0000 mon.smithi093 (mon.0) 501 : 2021-11-20T08:56:50.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:50 smithi093 conmon[33522]: audit [INF] from='client.? 172.21.15.93:0/1534599771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2021-11-20T08:56:50.973 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T08:56:50.985 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T08:56:50.986 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T08:56:51.607 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:51 smithi093 conmon[33522]: audit 2021-11-20T08:56:50.490108+0000 mon.smithi093 2021-11-20T08:56:51.608 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:51 smithi093 conmon[33522]: (mon.0) 502 : audit [INF] from='client.? 172.21.15.93:0/1534599771' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2021-11-20T08:56:51.608 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:51 smithi093 conmon[33522]: cluster 2021-11-20T08:56:50.490158+0000 mon.smithi093 (mon.0) 503 : cluster [DBG] 2021-11-20T08:56:51.608 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:51 smithi093 conmon[33522]: fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 2 up:standby 2021-11-20T08:56:51.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:51 smithi102 conmon[36472]: audit 2021-11-20T08:56:50.490108+0000 mon.smithi093 (mon.0) 502 : audit 2021-11-20T08:56:51.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:51 smithi102 conmon[36472]: [INF] from='client.? 172.21.15.93:0/1534599771' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2021-11-20T08:56:51.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:51 smithi102 conmon[36472]: cluster 2021-11-20T08:56:50.490158+0000 mon.smithi093 (mon.0) 503 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 2 up:standby 2021-11-20T08:56:52.579 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 -- 172.21.15.93:0/2960177952 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 msgr2=0x7f2be80f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:52.579 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 --2- 172.21.15.93:0/2960177952 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be80f7c30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2bdc004660 tx=0x7f2bdc009b30).stop 2021-11-20T08:56:52.579 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 -- 172.21.15.93:0/2960177952 shutdown_connections 2021-11-20T08:56:52.580 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 --2- 172.21.15.93:0/2960177952 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be80f7c30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:52.580 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 --2- 172.21.15.93:0/2960177952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2be80f6530 0x7f2be80f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:52.580 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 -- 172.21.15.93:0/2960177952 >> 172.21.15.93:0/2960177952 conn(0x7f2be80f1cd0 msgr2=0x7f2be80f40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:52.580 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 -- 172.21.15.93:0/2960177952 shutdown_connections 2021-11-20T08:56:52.581 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.577+0000 7f2be7fff700 1 -- 172.21.15.93:0/2960177952 wait complete. 2021-11-20T08:56:52.581 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be7fff700 1 Processor -- start 2021-11-20T08:56:52.581 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be7fff700 1 -- start start 2021-11-20T08:56:52.581 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be7fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2be80f6530 0x7f2be806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:52.582 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be7fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:52.582 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be7fff700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be806a380 con 0x7f2be80f77d0 2021-11-20T08:56:52.582 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be7fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be806a4c0 con 0x7f2be80f6530 2021-11-20T08:56:52.582 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be67fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:52.583 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be67fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be8069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43074/0 (socket says 172.21.15.93:43074) 2021-11-20T08:56:52.584 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be67fc700 1 -- 172.21.15.93:0/1426751545 learned_addr learned my addr 172.21.15.93:0/1426751545 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:52.584 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.578+0000 7f2be6ffd700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2be80f6530 0x7f2be806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:52.586 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2be67fc700 1 -- 172.21.15.93:0/1426751545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2be80f6530 msgr2=0x7f2be806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:52.586 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2be67fc700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2be80f6530 0x7f2be806b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:52.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2be67fc700 1 -- 172.21.15.93:0/1426751545 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc005040 con 0x7f2be80f77d0 2021-11-20T08:56:52.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2be67fc700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be8069e20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f2bdc009f50 tx=0x7f2bdc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:52.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2bcf7fe700 1 -- 172.21.15.93:0/1426751545 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bdc018070 con 0x7f2be80f77d0 2021-11-20T08:56:52.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be806a740 con 0x7f2be80f77d0 2021-11-20T08:56:52.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2bcf7fe700 1 -- 172.21.15.93:0/1426751545 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bdc013410 con 0x7f2be80f77d0 2021-11-20T08:56:52.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2bcf7fe700 1 -- 172.21.15.93:0/1426751545 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bdc01c3c0 con 0x7f2be80f77d0 2021-11-20T08:56:52.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.579+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be806aca0 con 0x7f2be80f77d0 2021-11-20T08:56:52.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.580+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be8045ad0 con 0x7f2be80f77d0 2021-11-20T08:56:52.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.581+0000 7f2bcf7fe700 1 -- 172.21.15.93:0/1426751545 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2bdc01d420 con 0x7f2be80f77d0 2021-11-20T08:56:52.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.581+0000 7f2bcf7fe700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2bd005c300 0x7f2bd005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:52.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.581+0000 7f2bcf7fe700 1 -- 172.21.15.93:0/1426751545 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2bdc077fe0 con 0x7f2be80f77d0 2021-11-20T08:56:52.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.581+0000 7f2be6ffd700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2bd005c300 0x7f2bd005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:52.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.582+0000 7f2be6ffd700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2bd005c300 0x7f2bd005e7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f2be806b730 tx=0x7f2bd800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:52.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.584+0000 7f2bcf7fe700 1 -- 172.21.15.93:0/1426751545 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bdc04ad10 con 0x7f2be80f77d0 2021-11-20T08:56:52.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: cluster 2021-11-20T08:56:51.497091+0000 2021-11-20T08:56:52.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: mon.smithi093 (mon.0) 504 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:standby 2021-11-20T08:56:52.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: cluster 2021-11-20T08:56:51.497141+0000 mon.smithi093 (mon.0) 505 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:boot 2021-11-20T08:56:52.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: cluster 2021-11-20T08:56:51.497192+0000 mon.smithi093 (mon.0) 506 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 3 up:standby 2021-11-20T08:56:52.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: audit 2021-11-20T08:56:51.497324+0000 mon.smithi093 (mon.0) 507 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:56:52.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: audit 2021-11-20T08:56:51.514748+0000 mon.smithi093 (mon.0) 508 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:52.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: audit 2021-11-20T08:56:51.515567+0000 mon.smithi093 (mon.0) 509 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:56:52.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:52 smithi093 conmon[33522]: cluster 2021-11-20T08:56:51.523432+0000 mgr.smithi093.qdikkm (mgr.14162) 189 : cluster [DBG] pgmap v152: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 4 op/s 2021-11-20T08:56:52.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.781+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2be8027a00 con 0x7f2be80f77d0 2021-11-20T08:56:52.781 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.781+0000 7f2bcf7fe700 1 -- 172.21.15.93:0/1426751545 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 10 v10) v1 ==== 76+0+1595 (secure 0 0 0) 0x7f2bdc04e330 con 0x7f2be80f77d0 2021-11-20T08:56:52.781 INFO:teuthology.orchestra.run.smithi093.stdout:e10 2021-11-20T08:56:52.781 INFO:teuthology.orchestra.run.smithi093.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T08:56:52.782 INFO:teuthology.orchestra.run.smithi093.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T08:56:52.782 INFO:teuthology.orchestra.run.smithi093.stdout:legacy client fscid: 1 2021-11-20T08:56:52.782 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:52.782 INFO:teuthology.orchestra.run.smithi093.stdout:Filesystem 'cephfs' (1) 2021-11-20T08:56:52.783 INFO:teuthology.orchestra.run.smithi093.stdout:fs_name cephfs 2021-11-20T08:56:52.783 INFO:teuthology.orchestra.run.smithi093.stdout:epoch 9 2021-11-20T08:56:52.783 INFO:teuthology.orchestra.run.smithi093.stdout:flags 12 2021-11-20T08:56:52.783 INFO:teuthology.orchestra.run.smithi093.stdout:created 2021-11-20T08:56:42.318132+0000 2021-11-20T08:56:52.784 INFO:teuthology.orchestra.run.smithi093.stdout:modified 2021-11-20T08:56:50.485290+0000 2021-11-20T08:56:52.784 INFO:teuthology.orchestra.run.smithi093.stdout:tableserver 0 2021-11-20T08:56:52.784 INFO:teuthology.orchestra.run.smithi093.stdout:root 0 2021-11-20T08:56:52.784 INFO:teuthology.orchestra.run.smithi093.stdout:session_timeout 60 2021-11-20T08:56:52.785 INFO:teuthology.orchestra.run.smithi093.stdout:session_autoclose 300 2021-11-20T08:56:52.785 INFO:teuthology.orchestra.run.smithi093.stdout:max_file_size 1099511627776 2021-11-20T08:56:52.785 INFO:teuthology.orchestra.run.smithi093.stdout:required_client_features {} 2021-11-20T08:56:52.785 INFO:teuthology.orchestra.run.smithi093.stdout:last_failure 0 2021-11-20T08:56:52.785 INFO:teuthology.orchestra.run.smithi093.stdout:last_failure_osd_epoch 0 2021-11-20T08:56:52.786 INFO:teuthology.orchestra.run.smithi093.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T08:56:52.786 INFO:teuthology.orchestra.run.smithi093.stdout:max_mds 1 2021-11-20T08:56:52.786 INFO:teuthology.orchestra.run.smithi093.stdout:in 0 2021-11-20T08:56:52.786 INFO:teuthology.orchestra.run.smithi093.stdout:up {0=24257} 2021-11-20T08:56:52.787 INFO:teuthology.orchestra.run.smithi093.stdout:failed 2021-11-20T08:56:52.787 INFO:teuthology.orchestra.run.smithi093.stdout:damaged 2021-11-20T08:56:52.787 INFO:teuthology.orchestra.run.smithi093.stdout:stopped 2021-11-20T08:56:52.787 INFO:teuthology.orchestra.run.smithi093.stdout:data_pools [3] 2021-11-20T08:56:52.788 INFO:teuthology.orchestra.run.smithi093.stdout:metadata_pool 2 2021-11-20T08:56:52.788 INFO:teuthology.orchestra.run.smithi093.stdout:inline_data disabled 2021-11-20T08:56:52.788 INFO:teuthology.orchestra.run.smithi093.stdout:balancer 2021-11-20T08:56:52.788 INFO:teuthology.orchestra.run.smithi093.stdout:standby_count_wanted 1 2021-11-20T08:56:52.788 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.cephfs.smithi102.lhrlmh{0:24257} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558]] 2021-11-20T08:56:52.789 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:52.789 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:52.789 INFO:teuthology.orchestra.run.smithi093.stdout:Standby daemons: 2021-11-20T08:56:52.790 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:52.790 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.cephfs.smithi093.ljzugc{-1:14428} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877]] 2021-11-20T08:56:52.790 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.cephfs.smithi093.zrryyo{-1:14438} state up:standby seq 1 addr [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814]] 2021-11-20T08:56:52.790 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.cephfs.smithi102.hjodmi{-1:24265} state up:standby seq 1 addr [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428]] 2021-11-20T08:56:52.791 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2bd005c300 msgr2=0x7f2bd005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:52.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2bd005c300 0x7f2bd005e7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f2be806b730 tx=0x7f2bd800b040).stop 2021-11-20T08:56:52.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 msgr2=0x7f2be8069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:52.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be8069e20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f2bdc009f50 tx=0x7f2bdc009b30).stop 2021-11-20T08:56:52.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 shutdown_connections 2021-11-20T08:56:52.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2bd005c300 0x7f2bd005e7c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:52.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2be80f6530 0x7f2be806b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:52.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 --2- 172.21.15.93:0/1426751545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2be80f77d0 0x7f2be8069e20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:52.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 >> 172.21.15.93:0/1426751545 conn(0x7f2be80f1cd0 msgr2=0x7f2be80fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:52.794 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 shutdown_connections 2021-11-20T08:56:52.794 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:52.783+0000 7f2be7fff700 1 -- 172.21.15.93:0/1426751545 wait complete. 2021-11-20T08:56:52.794 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 10 2021-11-20T08:56:52.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:52 smithi102 conmon[36472]: cluster 2021-11-20T08:56:51.497091+0000 mon.smithi093 (mon.0) 504 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:standby 2021-11-20T08:56:52.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:52 smithi102 conmon[36472]: cluster 2021-11-20T08:56:51.497141+0000 mon.smithi093 (mon.0) 505 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:boot 2021-11-20T08:56:52.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:52 smithi102 conmon[36472]: cluster 2021-11-20T08:56:51.497192+0000 mon.smithi093 (mon.0) 506 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 3 up:standby 2021-11-20T08:56:52.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:52 smithi102 conmon[36472]: audit 2021-11-20T08:56:51.497324+0000 mon.smithi093 (mon.0) 507 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:56:52.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:52 smithi102 conmon[36472]: audit 2021-11-20T08:56:51.514748+0000 mon.smithi093 (mon.0) 508 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:52.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:52 smithi102 conmon[36472]: audit 2021-11-20T08:56:51.515567+0000 mon.smithi093 (mon.0) 509 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:56:52.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:52 smithi102 conmon[36472]: cluster 2021-11-20T08:56:51.523432+0000 mgr.smithi093.qdikkm (mgr.14162) 189 : cluster [DBG] pgmap v152: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 4 op/s 2021-11-20T08:56:53.258 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2021-11-20T08:56:53.854 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:53 smithi093 conmon[33522]: audit 2021-11-20T08:56:52.782395+0000 mon.smithi093 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.93:0/1426751545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T08:56:53.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:53 smithi102 conmon[36472]: audit 2021-11-20T08:56:52.782395+0000 mon.smithi093 ( 2021-11-20T08:56:53.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:53 smithi102 conmon[36472]: mon.0) 510 : audit [DBG] from='client.? 172.21.15.93:0/1426751545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T08:56:54.775 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:54 smithi093 conmon[33522]: cluster 2021-11-20T08:56:53.517784+0000 mon.smithi093 ( 2021-11-20T08:56:54.775 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:54 smithi093 conmon[33522]: mon.0) 511 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:standby 2021-11-20T08:56:54.775 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:54 smithi093 conmon[33522]: cluster 2021-11-20T08:56:53.517844+0000 mon.smithi093 (mon.0) 512 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 3 up:standby 2021-11-20T08:56:54.775 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:54 smithi093 conmon[33522]: cluster 2021-11-20T08:56:53.523760+0000 mgr.smithi093.qdikkm (mgr.14162) 190 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 3 op/s 2021-11-20T08:56:54.776 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:54 smithi093 conmon[33522]: audit 2021-11-20T08:56:54.286218+0000 mon.smithi093 (mon.0) 513 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:54.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: cluster 2021-11-20T08:56:53.517784+0000 mon.smithi093 (mon.0 2021-11-20T08:56:54.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: ) 511 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:standby 2021-11-20T08:56:54.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: cluster 2021-11-20T08:56:53.517844+0000 mon.smithi093 (mon.0) 512 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 3 up:standby 2021-11-20T08:56:54.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: cluster 2021-11-20T08:56:53.523760+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:56:54.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: .14162) 190 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 3 op/s 2021-11-20T08:56:54.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:56:54.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: 20T08:56:54.286218+0000 mon.smithi093 (mon 2021-11-20T08:56:54.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:54 smithi102 conmon[36472]: .0) 513 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:55.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 -- 172.21.15.93:0/3790967945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406b9c0 msgr2=0x7f13c4069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:55.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/3790967945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406b9c0 0x7f13c4069ec0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f13b8004660 tx=0x7f13b8009b30).stop 2021-11-20T08:56:55.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 -- 172.21.15.93:0/3790967945 shutdown_connections 2021-11-20T08:56:55.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/3790967945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406a3b0 0x7f13c406a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:55.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/3790967945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406b9c0 0x7f13c4069ec0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:55.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 -- 172.21.15.93:0/3790967945 >> 172.21.15.93:0/3790967945 conn(0x7f13c40f1e40 msgr2=0x7f13c40f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:55.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 -- 172.21.15.93:0/3790967945 shutdown_connections 2021-11-20T08:56:55.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.076+0000 7f13cb2c4700 1 -- 172.21.15.93:0/3790967945 wait complete. 2021-11-20T08:56:55.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13cb2c4700 1 Processor -- start 2021-11-20T08:56:55.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13cb2c4700 1 -- start start 2021-11-20T08:56:55.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13cb2c4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406a3b0 0x7f13c410d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:55.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13cb2c4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406b9c0 0x7f13c410da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:55.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13cb2c4700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c410dfc0 con 0x7f13c406a3b0 2021-11-20T08:56:55.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13cb2c4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c410e100 con 0x7f13c406b9c0 2021-11-20T08:56:55.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c885f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406b9c0 0x7f13c410da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:55.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c9060700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406a3b0 0x7f13c410d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:55.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c885f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406b9c0 0x7f13c410da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.93:60242/0 (socket says 172.21.15.93:60242) 2021-11-20T08:56:55.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c885f700 1 -- 172.21.15.93:0/2531213580 learned_addr learned my addr 172.21.15.93:0/2531213580 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:55.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c9060700 1 -- 172.21.15.93:0/2531213580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406b9c0 msgr2=0x7f13c410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:55.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c9060700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406b9c0 0x7f13c410da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:55.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c9060700 1 -- 172.21.15.93:0/2531213580 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b8005040 con 0x7f13c406a3b0 2021-11-20T08:56:55.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.077+0000 7f13c885f700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406b9c0 0x7f13c410da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T08:56:55.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.078+0000 7f13c9060700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406a3b0 0x7f13c410d520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f13b8000c00 tx=0x7f13b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:55.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.078+0000 7f13b5ffb700 1 -- 172.21.15.93:0/2531213580 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13b8019070 con 0x7f13c406a3b0 2021-11-20T08:56:55.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.078+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c410ea40 con 0x7f13c406a3b0 2021-11-20T08:56:55.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.078+0000 7f13b5ffb700 1 -- 172.21.15.93:0/2531213580 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13b8007e70 con 0x7f13c406a3b0 2021-11-20T08:56:55.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.078+0000 7f13b5ffb700 1 -- 172.21.15.93:0/2531213580 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13b8014c40 con 0x7f13c406a3b0 2021-11-20T08:56:55.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.078+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c410efa0 con 0x7f13c406a3b0 2021-11-20T08:56:55.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.079+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c4045ad0 con 0x7f13c406a3b0 2021-11-20T08:56:55.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.079+0000 7f13b5ffb700 1 -- 172.21.15.93:0/2531213580 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f13b8025070 con 0x7f13c406a3b0 2021-11-20T08:56:55.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.080+0000 7f13b5ffb700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f13b005c2b0 0x7f13b005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:55.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.080+0000 7f13b5ffb700 1 -- 172.21.15.93:0/2531213580 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f13b8076d50 con 0x7f13c406a3b0 2021-11-20T08:56:55.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.080+0000 7f13c885f700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f13b005c2b0 0x7f13b005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:55.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.080+0000 7f13c885f700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f13b005c2b0 0x7f13b005e770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f13c0007a30 tx=0x7f13c000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:55.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.084+0000 7f13b5ffb700 1 -- 172.21.15.93:0/2531213580 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13b8049a10 con 0x7f13c406a3b0 2021-11-20T08:56:55.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.277+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13c4027a00 con 0x7f13c406a3b0 2021-11-20T08:56:55.277 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.278+0000 7f13b5ffb700 1 -- 172.21.15.93:0/2531213580 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2933 (secure 0 0 0) 0x7f13c4027a00 con 0x7f13c406a3b0 2021-11-20T08:56:55.278 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f13b005c2b0 msgr2=0x7f13b005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:55.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f13b005c2b0 0x7f13b005e770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f13c0007a30 tx=0x7f13c000b040).stop 2021-11-20T08:56:55.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406a3b0 msgr2=0x7f13c410d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:55.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406a3b0 0x7f13c410d520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f13b8000c00 tx=0x7f13b800b040).stop 2021-11-20T08:56:55.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 shutdown_connections 2021-11-20T08:56:55.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f13b005c2b0 0x7f13b005e770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:55.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13c406a3b0 0x7f13c410d520 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:55.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 --2- 172.21.15.93:0/2531213580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c406b9c0 0x7f13c410da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:55.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 >> 172.21.15.93:0/2531213580 conn(0x7f13c40f1e40 msgr2=0x7f13c40f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:55.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 shutdown_connections 2021-11-20T08:56:55.282 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:55.280+0000 7f13cb2c4700 1 -- 172.21.15.93:0/2531213580 wait complete. 2021-11-20T08:56:55.282 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 11 2021-11-20T08:56:55.292 INFO:teuthology.orchestra.run.smithi093.stdout:true 2021-11-20T08:56:55.739 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11458b34-49df-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json mds versions | jq -e ". | add == 4"' 2021-11-20T08:56:55.764 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:55 smithi093 conmon[33522]: audit 2021-11-20T08:56:55.278769+0000 mon.smithi093 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.93:0/2531213580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:56:55.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:55 smithi102 conmon[36472]: audit 2021-11-20T08:56:55.278769+0000 mon.smithi093 (mon.0) 514 : audit 2021-11-20T08:56:55.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:55 smithi102 conmon[36472]: [DBG] from='client.? 172.21.15.93:0/2531213580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:56:56.798 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:56 smithi093 conmon[33522]: cluster 2021-11-20T08:56:55. 2021-11-20T08:56:56.799 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:56 smithi093 conmon[33522]: 524142+0000 mgr.smithi093.qdikkm (mgr.14162) 191 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 2 op/s 2021-11-20T08:56:56.799 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:56 smithi093 conmon[33522]: cluster 2021-11-20T08:56:55.531756+0000 mon.smithi093 (mon.0) 515 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:standby 2021-11-20T08:56:56.799 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:56 smithi093 conmon[33522]: cluster 2021-11-20T08:56:55.531794+0000 mon.smithi093 (mon.0) 516 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 3 up:standby 2021-11-20T08:56:56.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:56 smithi102 conmon[36472]: cluster 2021-11-20T08:56:55.524142+0000 mgr.smithi093.qdikkm (mgr.14162) 191 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 2 op/s 2021-11-20T08:56:56.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:56 smithi102 conmon[36472]: 2021-11-20T08:56:56.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:56 smithi102 conmon[36472]: cluster 2021-11-20T08:56:55.531756+0000 mon.smithi093 (mon.0) 515 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:standby 2021-11-20T08:56:56.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:56 smithi102 conmon[36472]: cluster 2021-11-20T08:56:56.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:56 smithi102 conmon[36472]: 2021-11-20T08:56:55.531794+0000 mon.smithi093 (mon.0) 516 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi102.lhrlmh=up:active} 3 up:standby 2021-11-20T08:56:56.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:56 smithi102 conmon[36472]: 2021-11-20T08:56:57.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 -- 172.21.15.93:0/2190255486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f55c0 msgr2=0x7f8ff80f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:57.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/2190255486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f55c0 0x7f8ff80f59e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f8fec004660 tx=0x7f8fec009b30).stop 2021-11-20T08:56:57.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 -- 172.21.15.93:0/2190255486 shutdown_connections 2021-11-20T08:56:57.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/2190255486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff80f6860 0x7f8ff80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:57.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/2190255486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f55c0 0x7f8ff80f59e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:57.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 -- 172.21.15.93:0/2190255486 >> 172.21.15.93:0/2190255486 conn(0x7f8ff80f0d40 msgr2=0x7f8ff80f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:57.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 -- 172.21.15.93:0/2190255486 shutdown_connections 2021-11-20T08:56:57.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.218+0000 7f8ffeabb700 1 -- 172.21.15.93:0/2190255486 wait complete. 2021-11-20T08:56:57.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffeabb700 1 Processor -- start 2021-11-20T08:56:57.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffeabb700 1 -- start start 2021-11-20T08:56:57.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffeabb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff80f55c0 0x7f8ff8109c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:57.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffeabb700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f6860 0x7f8ff810c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:57.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffeabb700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80603c0 con 0x7f8ff80f6860 2021-11-20T08:56:57.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffeabb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff8060500 con 0x7f8ff80f55c0 2021-11-20T08:56:57.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ff7fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f6860 0x7f8ff810c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:57.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ff7fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f6860 0x7f8ff810c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43092/0 (socket says 172.21.15.93:43092) 2021-11-20T08:56:57.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ff7fff700 1 -- 172.21.15.93:0/3553735744 learned_addr learned my addr 172.21.15.93:0/3553735744 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:57.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffc857700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff80f55c0 0x7f8ff8109c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:57.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ff7fff700 1 -- 172.21.15.93:0/3553735744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff80f55c0 msgr2=0x7f8ff8109c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:57.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ff7fff700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff80f55c0 0x7f8ff8109c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:57.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ff7fff700 1 -- 172.21.15.93:0/3553735744 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec005040 con 0x7f8ff80f6860 2021-11-20T08:56:57.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ffc857700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff80f55c0 0x7f8ff8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T08:56:57.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.219+0000 7f8ff7fff700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f6860 0x7f8ff810c130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8fe400e7f0 tx=0x7f8fe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:57.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.220+0000 7f8ff57fa700 1 -- 172.21.15.93:0/3553735744 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fe40092e0 con 0x7f8ff80f6860 2021-11-20T08:56:57.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.220+0000 7f8ff57fa700 1 -- 172.21.15.93:0/3553735744 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fe400f040 con 0x7f8ff80f6860 2021-11-20T08:56:57.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.220+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff810c6e0 con 0x7f8ff80f6860 2021-11-20T08:56:57.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.220+0000 7f8ff57fa700 1 -- 172.21.15.93:0/3553735744 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fe40146c0 con 0x7f8ff80f6860 2021-11-20T08:56:57.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.220+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff810cc70 con 0x7f8ff80f6860 2021-11-20T08:56:57.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.221+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff80f49b0 con 0x7f8ff80f6860 2021-11-20T08:56:57.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.221+0000 7f8ff57fa700 1 -- 172.21.15.93:0/3553735744 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8fe401b030 con 0x7f8ff80f6860 2021-11-20T08:56:57.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.221+0000 7f8ff57fa700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8fe805c2b0 0x7f8fe805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:57.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.221+0000 7f8ff57fa700 1 -- 172.21.15.93:0/3553735744 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8fe4075b80 con 0x7f8ff80f6860 2021-11-20T08:56:57.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.221+0000 7f8ffc857700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8fe805c2b0 0x7f8fe805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:57.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.222+0000 7f8ffc857700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8fe805c2b0 0x7f8fe805e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8fec000c00 tx=0x7f8fec00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:57.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.225+0000 7f8ff57fa700 1 -- 172.21.15.93:0/3553735744 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fe4048890 con 0x7f8ff80f6860 2021-11-20T08:56:57.445 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.446+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f8ff8045ad0 con 0x7f8ff80f6860 2021-11-20T08:56:57.445 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.446+0000 7f8ff57fa700 1 -- 172.21.15.93:0/3553735744 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v12) v1 ==== 78+0+86 (secure 0 0 0) 0x7f8fe4019020 con 0x7f8ff80f6860 2021-11-20T08:56:57.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8fe805c2b0 msgr2=0x7f8fe805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:57.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8fe805c2b0 0x7f8fe805e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8fec000c00 tx=0x7f8fec00b040).stop 2021-11-20T08:56:57.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f6860 msgr2=0x7f8ff810c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:57.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f6860 0x7f8ff810c130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8fe400e7f0 tx=0x7f8fe400b040).stop 2021-11-20T08:56:57.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 shutdown_connections 2021-11-20T08:56:57.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8fe805c2b0 0x7f8fe805e770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:57.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ff80f55c0 0x7f8ff8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:57.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 --2- 172.21.15.93:0/3553735744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ff80f6860 0x7f8ff810c130 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:57.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 >> 172.21.15.93:0/3553735744 conn(0x7f8ff80f0d40 msgr2=0x7f8ff80fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:57.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 shutdown_connections 2021-11-20T08:56:57.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:57.448+0000 7f8ffeabb700 1 -- 172.21.15.93:0/3553735744 wait complete. 2021-11-20T08:56:57.460 INFO:teuthology.orchestra.run.smithi093.stdout:true 2021-11-20T08:56:57.812 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:57 smithi093 conmon[33522]: audit 2021-11-20T08:56:56.620049+0000 mon.smithi093 (mon.0) 517 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:56:57.812 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:57 smithi093 conmon[33522]: audit 2021-11-20T08:56:56.620228+0000 mon.smithi093 (mon.0) 518 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:56:57.812 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:57 smithi093 conmon[33522]: audit 2021-11-20T08:56:57.447425+0000 mon.smithi093 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.93:0/3553735744' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T08:56:57.813 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:57 smithi093 conmon[33522]: cluster 2021-11-20T08:56:57.524490+0000 mgr.smithi093.qdikkm (mgr.14162) 192 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 2 op/s 2021-11-20T08:56:57.852 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2021-11-20T08:56:57.878 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T08:56:57.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:57 smithi102 conmon[36472]: audit 2021-11-20T08:56:56.620049+0000 mon.smithi093 (mon.0) 517 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:56:57.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:57 smithi102 conmon[36472]: audit 2021-11-20T08:56:56.620228+0000 mon.smithi093 (mon.0) 518 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:56:57.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:57 smithi102 conmon[36472]: audit 2021-11-20T08:56:57.447425+0000 mon.smithi093 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.93:0/3553735744' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T08:56:57.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:57 smithi102 conmon[36472]: cluster 2021-11-20T08:56:57.524490+0000 mgr.smithi093.qdikkm (mgr.14162) 192 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 2 op/s 2021-11-20T08:56:58.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.047+0000 7fc08a3ca700 1 -- 172.21.15.93:0/3376767346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 msgr2=0x7fc0840f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.047+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/3376767346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840f59d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc074004660 tx=0x7fc074009b30).stop 2021-11-20T08:56:58.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.048+0000 7fc08a3ca700 1 -- 172.21.15.93:0/3376767346 shutdown_connections 2021-11-20T08:56:58.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.048+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/3376767346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0840f6850 0x7fc0840f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.048+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/3376767346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840f59d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.048+0000 7fc08a3ca700 1 -- 172.21.15.93:0/3376767346 >> 172.21.15.93:0/3376767346 conn(0x7fc0840f0d10 msgr2=0x7fc0840f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.048+0000 7fc08a3ca700 1 -- 172.21.15.93:0/3376767346 shutdown_connections 2021-11-20T08:56:58.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.048+0000 7fc08a3ca700 1 -- 172.21.15.93:0/3376767346 wait complete. 2021-11-20T08:56:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.048+0000 7fc08a3ca700 1 Processor -- start 2021-11-20T08:56:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc08a3ca700 1 -- start start 2021-11-20T08:56:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc08a3ca700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc08a3ca700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0840f6850 0x7fc08410bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc08a3ca700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08410c150 con 0x7fc0840f55b0 2021-11-20T08:56:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc08a3ca700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08410c290 con 0x7fc0840f6850 2021-11-20T08:56:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc083fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc083fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43098/0 (socket says 172.21.15.93:43098) 2021-11-20T08:56:58.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc083fff700 1 -- 172.21.15.93:0/2254653112 learned_addr learned my addr 172.21.15.93:0/2254653112 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:58.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc0837fe700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0840f6850 0x7fc08410bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc0837fe700 1 -- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 msgr2=0x7fc0840604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc0837fe700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840604d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc0837fe700 1 -- 172.21.15.93:0/2254653112 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074005040 con 0x7fc0840f6850 2021-11-20T08:56:58.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc083fff700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T08:56:58.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.049+0000 7fc0837fe700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0840f6850 0x7fc08410bc60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc07800e7c0 tx=0x7fc07800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:58.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.050+0000 7fc080ff9700 1 -- 172.21.15.93:0/2254653112 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0780092e0 con 0x7fc0840f6850 2021-11-20T08:56:58.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.050+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08410c4f0 con 0x7fc0840f6850 2021-11-20T08:56:58.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.050+0000 7fc080ff9700 1 -- 172.21.15.93:0/2254653112 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc07800f040 con 0x7fc0840f6850 2021-11-20T08:56:58.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.050+0000 7fc080ff9700 1 -- 172.21.15.93:0/2254653112 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc078014880 con 0x7fc0840f6850 2021-11-20T08:56:58.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.050+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08410cab0 con 0x7fc0840f6850 2021-11-20T08:56:58.059 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.051+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc084027a00 con 0x7fc0840f6850 2021-11-20T08:56:58.059 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.051+0000 7fc080ff9700 1 -- 172.21.15.93:0/2254653112 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc07801b030 con 0x7fc0840f6850 2021-11-20T08:56:58.059 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.051+0000 7fc080ff9700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc06c05c2a0 0x7fc06c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.059 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.052+0000 7fc080ff9700 1 -- 172.21.15.93:0/2254653112 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc078075cf0 con 0x7fc0840f6850 2021-11-20T08:56:58.060 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.052+0000 7fc083fff700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc06c05c2a0 0x7fc06c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.060 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.052+0000 7fc083fff700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc06c05c2a0 0x7fc06c05e760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc074005010 tx=0x7fc074016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:58.060 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.056+0000 7fc080ff9700 1 -- 172.21.15.93:0/2254653112 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0780489b0 con 0x7fc0840f6850 2021-11-20T08:56:58.216 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.218+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0840fdd40 con 0x7fc0840f6850 2021-11-20T08:56:58.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.219+0000 7fc080ff9700 1 -- 172.21.15.93:0/2254653112 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+14086 (secure 0 0 0) 0x7fc078019020 con 0x7fc0840f6850 2021-11-20T08:56:58.218 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:58.218 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":43,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","created":"2021-11-20T08:51:57.464444+0000","modified":"2021-11-20T08:56:44.326907+0000","last_up_change":"2021-11-20T08:55:49.151657+0000","last_in_change":"2021-11-20T08:55:32.747237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T08:55:09.134409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T08:56:41.253240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T08:56:41.551098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b62eaab5-818e-4739-98c3-d386425d4837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6803","nonce":1548414389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6805","nonce":1548414389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6809","nonce":1548414389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6807","nonce":1548414389}]},"public_addr":"172.21.15.93:6803/1548414389","cluster_addr":"172.21.15.93:6805/1548414389","heartbeat_back_addr":"172.21.15.93:6809/1548414389","heartbeat_front_addr":"172.21.15.93:6807/1548414389","state":["exists","up"]},{"osd":1,"uuid":"a46f3336-3bc4-4736-9095-c4d72e662323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6811","nonce":3785015356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6813","nonce":3785015356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6817","nonce":3785015356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6815","nonce":3785015356}]},"public_addr":"172.21.15.93:6811/3785015356","cluster_addr":"172.21.15.93:6813/3785015356","heartbeat_back_addr":"172.21.15.93:6817/3785015356","heartbeat_front_addr":"172.21.15.93:6815/3785015356","state":["exists","up"]},{"osd":2,"uuid":"72096543-af61-429d-a8d1-a77726a1f3eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6819","nonce":82486730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6821","nonce":82486730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6825","nonce":82486730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6823","nonce":82486730}]},"public_addr":"172.21.15.93:6819/82486730","cluster_addr":"172.21.15.93:6821/82486730","heartbeat_back_addr":"172.21.15.93:6825/82486730","heartbeat_front_addr":"172.21.15.93:6823/82486730","state":["exists","up"]},{"osd":3,"uuid":"37ba4275-7023-4f74-ba1f-7ad2f864476f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6801","nonce":1093924688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6803","nonce":1093924688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6807","nonce":1093924688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6805","nonce":1093924688}]},"public_addr":"172.21.15.102:6801/1093924688","cluster_addr":"172.21.15.102:6803/1093924688","heartbeat_back_addr":"172.21.15.102:6807/1093924688","heartbeat_front_addr":"172.21.15.102:6805/1093924688","state":["exists","up"]},{"osd":4,"uuid":"c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6809","nonce":2700333748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6811","nonce":2700333748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6815","nonce":2700333748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6813","nonce":2700333748}]},"public_addr":"172.21.15.102:6809/2700333748","cluster_addr":"172.21.15.102:6811/2700333748","heartbeat_back_addr":"172.21.15.102:6815/2700333748","heartbeat_front_addr":"172.21.15.102:6813/2700333748","state":["exists","up"]},{"osd":5,"uuid":"fab60808-4ace-4f26-9c42-f72340134a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6817","nonce":4083093586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6819","nonce":4083093586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6823","nonce":4083093586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6821","nonce":4083093586}]},"public_addr":"172.21.15.102:6817/4083093586","cluster_addr":"172.21.15.102:6819/4083093586","heartbeat_back_addr":"172.21.15.102:6823/4083093586","heartbeat_front_addr":"172.21.15.102:6821/4083093586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:21.010676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:41.314338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:57.961372+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:13.733626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:30.250474+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:47.331557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:0/3689853676":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/3631043468":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/485709266":"2021-11-21T08:52:37.986811+0000","172.21.15.93:6800/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1880974768":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1518372611":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/2364704567":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/1580490298":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6801/1580490298":"2021-11-21T08:52:15.847551+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T08:56:58.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.220+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc06c05c2a0 msgr2=0x7fc06c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.220+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc06c05c2a0 0x7fc06c05e760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc074005010 tx=0x7fc074016040).stop 2021-11-20T08:56:58.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.220+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0840f6850 msgr2=0x7fc08410bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.220+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0840f6850 0x7fc08410bc60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc07800e7c0 tx=0x7fc07800b040).stop 2021-11-20T08:56:58.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.221+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 shutdown_connections 2021-11-20T08:56:58.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.221+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc06c05c2a0 0x7fc06c05e760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.221+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0840f55b0 0x7fc0840604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.221+0000 7fc08a3ca700 1 --2- 172.21.15.93:0/2254653112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0840f6850 0x7fc08410bc60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.221+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 >> 172.21.15.93:0/2254653112 conn(0x7fc0840f0d10 msgr2=0x7fc0840fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:58.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.221+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 shutdown_connections 2021-11-20T08:56:58.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.221+0000 7fc08a3ca700 1 -- 172.21.15.93:0/2254653112 wait complete. 2021-11-20T08:56:58.231 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T08:55:09.134409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T08:56:41.253240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T08:56:41.551098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T08:56:58.232 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T08:56:58.373 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 -- 172.21.15.93:0/3932666506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f6690 msgr2=0x7f76680f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.373 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 --2- 172.21.15.93:0/3932666506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f6690 0x7f76680f6af0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f7658004660 tx=0x7f7658009b30).stop 2021-11-20T08:56:58.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 -- 172.21.15.93:0/3932666506 shutdown_connections 2021-11-20T08:56:58.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 --2- 172.21.15.93:0/3932666506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f6690 0x7f76680f6af0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 --2- 172.21.15.93:0/3932666506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76680f53f0 0x7f76680f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 -- 172.21.15.93:0/3932666506 >> 172.21.15.93:0/3932666506 conn(0x7f76680f0b30 msgr2=0x7f76680f2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:58.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 -- 172.21.15.93:0/3932666506 shutdown_connections 2021-11-20T08:56:58.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.371+0000 7f766ea9d700 1 -- 172.21.15.93:0/3932666506 wait complete. 2021-11-20T08:56:58.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766ea9d700 1 Processor -- start 2021-11-20T08:56:58.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766ea9d700 1 -- start start 2021-11-20T08:56:58.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766ea9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f53f0 0x7f7668106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766ea9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76680f6690 0x7f7668107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766ea9d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7668107880 con 0x7f76680f53f0 2021-11-20T08:56:58.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766ea9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76681080c0 con 0x7f76680f6690 2021-11-20T08:56:58.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766da9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f53f0 0x7f7668106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766da9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f53f0 0x7f7668106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43108/0 (socket says 172.21.15.93:43108) 2021-11-20T08:56:58.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766da9b700 1 -- 172.21.15.93:0/4153059570 learned_addr learned my addr 172.21.15.93:0/4153059570 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:58.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766d29a700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76680f6690 0x7f7668107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766da9b700 1 -- 172.21.15.93:0/4153059570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76680f6690 msgr2=0x7f7668107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.378 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766da9b700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76680f6690 0x7f7668107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.378 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766da9b700 1 -- 172.21.15.93:0/4153059570 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658005040 con 0x7f76680f53f0 2021-11-20T08:56:58.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f766da9b700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f53f0 0x7f7668106d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7664002700 tx=0x7f7664009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:58.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.372+0000 7f765e7fc700 1 -- 172.21.15.93:0/4153059570 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76640079c0 con 0x7f76680f53f0 2021-11-20T08:56:58.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.373+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76681083a0 con 0x7f76680f53f0 2021-11-20T08:56:58.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.373+0000 7f765e7fc700 1 -- 172.21.15.93:0/4153059570 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f766400f040 con 0x7f76680f53f0 2021-11-20T08:56:58.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.373+0000 7f765e7fc700 1 -- 172.21.15.93:0/4153059570 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7664014850 con 0x7f76680f53f0 2021-11-20T08:56:58.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.373+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76681088e0 con 0x7f76680f53f0 2021-11-20T08:56:58.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.373+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76680f47e0 con 0x7f76680f53f0 2021-11-20T08:56:58.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.374+0000 7f765e7fc700 1 -- 172.21.15.93:0/4153059570 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f766401b060 con 0x7f76680f53f0 2021-11-20T08:56:58.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.374+0000 7f765e7fc700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f765405c2b0 0x7f765405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.374+0000 7f765e7fc700 1 -- 172.21.15.93:0/4153059570 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f7664075de0 con 0x7f76680f53f0 2021-11-20T08:56:58.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.374+0000 7f766d29a700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f765405c2b0 0x7f765405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.374+0000 7f766d29a700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f765405c2b0 0x7f765405e770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7668106b20 tx=0x7f7658009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:58.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.378+0000 7f765e7fc700 1 -- 172.21.15.93:0/4153059570 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7664048aa0 con 0x7f76680f53f0 2021-11-20T08:56:58.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.536+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7668027b50 con 0x7f76680f53f0 2021-11-20T08:56:58.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.536+0000 7f765e7fc700 1 -- 172.21.15.93:0/4153059570 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v43) v1 ==== 110+0+10 (secure 0 0 0) 0x7f766404c0c0 con 0x7f76680f53f0 2021-11-20T08:56:58.537 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 1 2021-11-20T08:56:58.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f765405c2b0 msgr2=0x7f765405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f765405c2b0 0x7f765405e770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7668106b20 tx=0x7f7658009b30).stop 2021-11-20T08:56:58.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f53f0 msgr2=0x7f7668106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f53f0 0x7f7668106d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7664002700 tx=0x7f7664009e00).stop 2021-11-20T08:56:58.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 shutdown_connections 2021-11-20T08:56:58.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f765405c2b0 0x7f765405e770 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680f53f0 0x7f7668106d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 --2- 172.21.15.93:0/4153059570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76680f6690 0x7f7668107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 >> 172.21.15.93:0/4153059570 conn(0x7f76680f0b30 msgr2=0x7f76680fa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:58.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 shutdown_connections 2021-11-20T08:56:58.543 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.538+0000 7f766ea9d700 1 -- 172.21.15.93:0/4153059570 wait complete. 2021-11-20T08:56:58.550 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T08:56:58.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.684+0000 7f4325ba1700 1 -- 172.21.15.93:0/2561435395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f6670 msgr2=0x7f43200f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.684+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2561435395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f6670 0x7f43200f6ad0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f430c004660 tx=0x7f430c009b30).stop 2021-11-20T08:56:58.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.684+0000 7f4325ba1700 1 -- 172.21.15.93:0/2561435395 shutdown_connections 2021-11-20T08:56:58.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.684+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2561435395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f6670 0x7f43200f6ad0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.684+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2561435395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43200f53d0 0x7f43200f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.684+0000 7f4325ba1700 1 -- 172.21.15.93:0/2561435395 >> 172.21.15.93:0/2561435395 conn(0x7f43200f0b30 msgr2=0x7f43200f2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:58.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 -- 172.21.15.93:0/2561435395 shutdown_connections 2021-11-20T08:56:58.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 -- 172.21.15.93:0/2561435395 wait complete. 2021-11-20T08:56:58.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 Processor -- start 2021-11-20T08:56:58.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 -- start start 2021-11-20T08:56:58.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f53d0 0x7f4320106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43200f6670 0x7f4320107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4320107880 con 0x7f43200f53d0 2021-11-20T08:56:58.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4325ba1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4320108090 con 0x7f43200f6670 2021-11-20T08:56:58.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.685+0000 7f4324b9f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f53d0 0x7f4320106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4324b9f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f53d0 0x7f4320106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43118/0 (socket says 172.21.15.93:43118) 2021-11-20T08:56:58.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4324b9f700 1 -- 172.21.15.93:0/2885055880 learned_addr learned my addr 172.21.15.93:0/2885055880 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:58.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f431ffff700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43200f6670 0x7f4320107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4324b9f700 1 -- 172.21.15.93:0/2885055880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43200f6670 msgr2=0x7f4320107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4324b9f700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43200f6670 0x7f4320107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4324b9f700 1 -- 172.21.15.93:0/2885055880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c005040 con 0x7f43200f53d0 2021-11-20T08:56:58.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4324b9f700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f53d0 0x7f4320106d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f431400e7c0 tx=0x7f431400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:58.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f431d7fa700 1 -- 172.21.15.93:0/2885055880 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43140092e0 con 0x7f43200f53d0 2021-11-20T08:56:58.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4320108310 con 0x7f43200f53d0 2021-11-20T08:56:58.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f431d7fa700 1 -- 172.21.15.93:0/2885055880 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f431400f040 con 0x7f43200f53d0 2021-11-20T08:56:58.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f431d7fa700 1 -- 172.21.15.93:0/2885055880 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43140147c0 con 0x7f43200f53d0 2021-11-20T08:56:58.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.686+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43201088d0 con 0x7f43200f53d0 2021-11-20T08:56:58.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.687+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43200f47c0 con 0x7f43200f53d0 2021-11-20T08:56:58.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.688+0000 7f431d7fa700 1 -- 172.21.15.93:0/2885055880 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f431401b030 con 0x7f43200f53d0 2021-11-20T08:56:58.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.688+0000 7f431d7fa700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f431005c2b0 0x7f431005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:58.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.688+0000 7f431d7fa700 1 -- 172.21.15.93:0/2885055880 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4314075e50 con 0x7f43200f53d0 2021-11-20T08:56:58.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.688+0000 7f431ffff700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f431005c2b0 0x7f431005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:58.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.688+0000 7f431ffff700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f431005c2b0 0x7f431005e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f430c005010 tx=0x7f430c016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:58.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.692+0000 7f431d7fa700 1 -- 172.21.15.93:0/2885055880 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4314048b10 con 0x7f43200f53d0 2021-11-20T08:56:58.861 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.861+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f4320045ad0 con 0x7f43200f53d0 2021-11-20T08:56:58.862 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.862+0000 7f431d7fa700 1 -- 172.21.15.93:0/2885055880 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7f4314019070 con 0x7f43200f53d0 2021-11-20T08:56:58.862 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 32 2021-11-20T08:56:58.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f431005c2b0 msgr2=0x7f431005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f431005c2b0 0x7f431005e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f430c005010 tx=0x7f430c016040).stop 2021-11-20T08:56:58.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f53d0 msgr2=0x7f4320106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:58.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f53d0 0x7f4320106d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f431400e7c0 tx=0x7f431400b040).stop 2021-11-20T08:56:58.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 shutdown_connections 2021-11-20T08:56:58.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f431005c2b0 0x7f431005e770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43200f53d0 0x7f4320106d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 --2- 172.21.15.93:0/2885055880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43200f6670 0x7f4320107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:58.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 >> 172.21.15.93:0/2885055880 conn(0x7f43200f0b30 msgr2=0x7f432005f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:58.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 shutdown_connections 2021-11-20T08:56:58.867 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:58.864+0000 7f4325ba1700 1 -- 172.21.15.93:0/2885055880 wait complete. 2021-11-20T08:56:58.874 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:58 smithi093 conmon[33522]: audit 2021-11-20T08:56:57 2021-11-20T08:56:58.874 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:58 smithi093 conmon[33522]: .712152+0000 mon.smithi093 (mon.0) 520 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:58.874 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:58 smithi093 conmon[33522]: audit 2021-11-20T08:56:57.715311+0000 mon.smithi093 (mon.0) 521 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:58.875 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:58 smithi093 conmon[33522]: audit 2021-11-20T08:56:57.718360+0000 mon.smithi093 (mon.0) 522 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:56:58.875 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:58 smithi093 conmon[33522]: audit 2021-11-20T08:56:58.219272+0000 mon.smithi102 (mon.1) 15 : audit [DBG] from='client.? 172.21.15.93:0/2254653112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:56:58.875 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:58 smithi093 conmon[33522]: audit 2021-11-20T08:56:58.537135+0000 mon.smithi093 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.93:0/4153059570' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:56:58.877 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T08:56:59.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 -- 172.21.15.93:0/2713410595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 msgr2=0x7f2e2c0f4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 --2- 172.21.15.93:0/2713410595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f4df0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2e240099e0 tx=0x7f2e24004d10).stop 2021-11-20T08:56:59.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 -- 172.21.15.93:0/2713410595 shutdown_connections 2021-11-20T08:56:59.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 --2- 172.21.15.93:0/2713410595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e2c0f5370 0x7f2e2c0f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 --2- 172.21.15.93:0/2713410595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f4df0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 -- 172.21.15.93:0/2713410595 >> 172.21.15.93:0/2713410595 conn(0x7f2e2c0f04d0 msgr2=0x7f2e2c0f2910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:59.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 -- 172.21.15.93:0/2713410595 shutdown_connections 2021-11-20T08:56:59.007 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.003+0000 7f2e32b38700 1 -- 172.21.15.93:0/2713410595 wait complete. 2021-11-20T08:56:59.007 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e32b38700 1 Processor -- start 2021-11-20T08:56:59.007 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e32b38700 1 -- start start 2021-11-20T08:56:59.007 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e32b38700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:59.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e32b38700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e2c0f5370 0x7f2e2c0f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:59.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e32b38700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0f7950 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e32b38700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0f5ef0 con 0x7f2e2c0f5370 2021-11-20T08:56:59.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e308d4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:59.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e308d4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43124/0 (socket says 172.21.15.93:43124) 2021-11-20T08:56:59.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e308d4700 1 -- 172.21.15.93:0/1177491017 learned_addr learned my addr 172.21.15.93:0/1177491017 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:59.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e2bfff700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e2c0f5370 0x7f2e2c0f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:59.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e308d4700 1 -- 172.21.15.93:0/1177491017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e2c0f5370 msgr2=0x7f2e2c0f5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.004+0000 7f2e308d4700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e2c0f5370 0x7f2e2c0f5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.005+0000 7f2e308d4700 1 -- 172.21.15.93:0/1177491017 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24009640 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.005+0000 7f2e308d4700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f73a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2e2400b3b0 tx=0x7f2e24004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:59.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.005+0000 7f2e297fa700 1 -- 172.21.15.93:0/1177491017 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e2400e030 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.005+0000 7f2e297fa700 1 -- 172.21.15.93:0/1177491017 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e24007d40 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.005+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c0f6110 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.005+0000 7f2e297fa700 1 -- 172.21.15.93:0/1177491017 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e24018070 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.005+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c0f6670 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.006+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c046330 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.006+0000 7f2e297fa700 1 -- 172.21.15.93:0/1177491017 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2e2401d420 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.012 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.006+0000 7f2e297fa700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2e1805c240 0x7f2e1805e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:59.012 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.006+0000 7f2e297fa700 1 -- 172.21.15.93:0/1177491017 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2e24077a30 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.012 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.006+0000 7f2e2bfff700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2e1805c240 0x7f2e1805e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:59.013 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.007+0000 7f2e2bfff700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2e1805c240 0x7f2e1805e700 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f2e14009fd0 tx=0x7f2e1400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:59.013 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.010+0000 7f2e297fa700 1 -- 172.21.15.93:0/1177491017 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e2404a6f0 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: audit 2021-11-20T08:56:57.712152+0000 mon.smithi093 (mon. 2021-11-20T08:56:59.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: 0) 520 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:59.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: audit 2021-11-20T08:56:57.715311+0000 mon.smithi093 (mon.0) 521 : 2021-11-20T08:56:59.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:56:59.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: audit 2021-11-20T08:56:59.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: 2021-11-20T08:56:57.718360 2021-11-20T08:56:59.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 522 : audit 2021-11-20T08:56:59.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:56:59.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: audit 2021-11-20T08:56:59.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: 2021-11-20T08:56: 2021-11-20T08:56:59.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: 58.219272+0000 mon.smithi102 (mon.1 2021-11-20T08:56:59.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: ) 15 : audit [DBG] from='client.? 172.21.15.93:0/2254653112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:56:59.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: audit 2021- 2021-11-20T08:56:59.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: 11-20T08:56:58.537135 2021-11-20T08:56:59.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 523 : 2021-11-20T08:56:59.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:58 smithi102 conmon[36472]: audit [DBG] from='client.? 172.21.15.93:0/4153059570' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:56:59.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.173+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2e2c027a00 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.173+0000 7f2e297fa700 1 -- 172.21.15.93:0/1177491017 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2e2404dd10 con 0x7f2e2c0f49d0 2021-11-20T08:56:59.172 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 32 2021-11-20T08:56:59.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2e1805c240 msgr2=0x7f2e1805e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2e1805c240 0x7f2e1805e700 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f2e14009fd0 tx=0x7f2e1400b040).stop 2021-11-20T08:56:59.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 msgr2=0x7f2e2c0f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f73a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2e2400b3b0 tx=0x7f2e24004d10).stop 2021-11-20T08:56:59.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 shutdown_connections 2021-11-20T08:56:59.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2e1805c240 0x7f2e1805e700 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e2c0f49d0 0x7f2e2c0f73a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 --2- 172.21.15.93:0/1177491017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e2c0f5370 0x7f2e2c0f5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 >> 172.21.15.93:0/1177491017 conn(0x7f2e2c0f04d0 msgr2=0x7f2e2c0fb540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:59.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 shutdown_connections 2021-11-20T08:56:59.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.175+0000 7f2e32b38700 1 -- 172.21.15.93:0/1177491017 wait complete. 2021-11-20T08:56:59.186 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:56:59.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 -- 172.21.15.93:0/877951745 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 msgr2=0x7f060c0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 --2- 172.21.15.93:0/877951745 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 0x7f060c0f6ca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f0600004660 tx=0x7f0600009b30).stop 2021-11-20T08:56:59.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 -- 172.21.15.93:0/877951745 shutdown_connections 2021-11-20T08:56:59.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 --2- 172.21.15.93:0/877951745 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 0x7f060c0f6ca0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 --2- 172.21.15.93:0/877951745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060c0f55a0 0x7f060c0f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 -- 172.21.15.93:0/877951745 >> 172.21.15.93:0/877951745 conn(0x7f060c0f0ce0 msgr2=0x7f060c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:59.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 -- 172.21.15.93:0/877951745 shutdown_connections 2021-11-20T08:56:59.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 -- 172.21.15.93:0/877951745 wait complete. 2021-11-20T08:56:59.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.306+0000 7f0612564700 1 Processor -- start 2021-11-20T08:56:59.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f0612564700 1 -- start start 2021-11-20T08:56:59.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f0612564700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060c0f55a0 0x7f060c105d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:59.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f0612564700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 0x7f060c106220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:59.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f0612564700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c1067d0 con 0x7f060c0f6840 2021-11-20T08:56:59.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f0612564700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c106910 con 0x7f060c0f55a0 2021-11-20T08:56:59.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 0x7f060c106220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:59.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 0x7f060c106220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43136/0 (socket says 172.21.15.93:43136) 2021-11-20T08:56:59.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060b7fe700 1 -- 172.21.15.93:0/2505128894 learned_addr learned my addr 172.21.15.93:0/2505128894 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:56:59.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060bfff700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060c0f55a0 0x7f060c105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:59.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060bfff700 1 -- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 msgr2=0x7f060c106220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060bfff700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 0x7f060c106220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060bfff700 1 -- 172.21.15.93:0/2505128894 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0600005040 con 0x7f060c0f55a0 2021-11-20T08:56:59.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.307+0000 7f060bfff700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060c0f55a0 0x7f060c105d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f060c0f66a0 tx=0x7f05fc00b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:59.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.308+0000 7f0608ff9700 1 -- 172.21.15.93:0/2505128894 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc0092e0 con 0x7f060c0f55a0 2021-11-20T08:56:59.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.308+0000 7f0608ff9700 1 -- 172.21.15.93:0/2505128894 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05fc00f040 con 0x7f060c0f55a0 2021-11-20T08:56:59.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.308+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c10b3b0 con 0x7f060c0f55a0 2021-11-20T08:56:59.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.308+0000 7f0608ff9700 1 -- 172.21.15.93:0/2505128894 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc0147a0 con 0x7f060c0f55a0 2021-11-20T08:56:59.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.308+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c10b970 con 0x7f060c0f55a0 2021-11-20T08:56:59.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.309+0000 7f0608ff9700 1 -- 172.21.15.93:0/2505128894 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f05fc01b030 con 0x7f060c0f55a0 2021-11-20T08:56:59.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.309+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c046330 con 0x7f060c0f55a0 2021-11-20T08:56:59.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.309+0000 7f0608ff9700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f05f405c2a0 0x7f05f405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:56:59.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.309+0000 7f0608ff9700 1 -- 172.21.15.93:0/2505128894 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f05fc075c60 con 0x7f060c0f55a0 2021-11-20T08:56:59.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.310+0000 7f060b7fe700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f05f405c2a0 0x7f05f405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:56:59.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.310+0000 7f060b7fe700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f05f405c2a0 0x7f05f405e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0600009d60 tx=0x7f0600016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:56:59.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.314+0000 7f0608ff9700 1 -- 172.21.15.93:0/2505128894 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05fc0489a0 con 0x7f060c0f55a0 2021-11-20T08:56:59.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.512+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f060c027a00 con 0x7f060c0f55a0 2021-11-20T08:56:59.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.512+0000 7f0608ff9700 1 -- 172.21.15.93:0/2505128894 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2932 (secure 0 0 0) 0x7f05fc019020 con 0x7f060c0f55a0 2021-11-20T08:56:59.512 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:56:59.512 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14428,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6827/2783208877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":2783208877},{"type":"v1","addr":"172.21.15.93:6827","nonce":2783208877}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":10},{"gid":14438,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/1976124814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":1976124814},{"type":"v1","addr":"172.21.15.93:6829","nonce":1976124814}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":12},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:56:50.485290+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.102:6825/1423063558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1423063558},{"type":"v1","addr":"172.21.15.102:6825","nonce":1423063558}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:56:59.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f05f405c2a0 msgr2=0x7f05f405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f05f405c2a0 0x7f05f405e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0600009d60 tx=0x7f0600016040).stop 2021-11-20T08:56:59.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060c0f55a0 msgr2=0x7f060c105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:56:59.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060c0f55a0 0x7f060c105d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f060c0f66a0 tx=0x7f05fc00b040).stop 2021-11-20T08:56:59.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 shutdown_connections 2021-11-20T08:56:59.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f05f405c2a0 0x7f05f405e760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060c0f55a0 0x7f060c105d30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 --2- 172.21.15.93:0/2505128894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f060c0f6840 0x7f060c106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:56:59.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 >> 172.21.15.93:0/2505128894 conn(0x7f060c0f0ce0 msgr2=0x7f060c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:56:59.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 shutdown_connections 2021-11-20T08:56:59.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:56:59.514+0000 7f0612564700 1 -- 172.21.15.93:0/2505128894 wait complete. 2021-11-20T08:56:59.516 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 12 2021-11-20T08:56:59.525 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 9, 'max_mds': 1, 'flags': 18} 2021-11-20T08:56:59.525 INFO:teuthology.run_tasks:Running task kclient... 2021-11-20T08:56:59.561 INFO:tasks.kclient:Mounting kernel clients... 2021-11-20T08:56:59.562 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2021-11-20T08:56:59.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:56:59.563 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2021-11-20T08:56:59.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:56:59.581 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link delete ceph-brx 2021-11-20T08:56:59.649 INFO:teuthology.orchestra.run.smithi093.stderr:Cannot find device "ceph-brx" 2021-11-20T08:56:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T08:56:59.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:56:59.651 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2021-11-20T08:56:59.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:56:59.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2021-11-20T08:56:59.739 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2021-11-20T08:56:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T08:56:59.741 INFO:tasks.kclient:client.0 config is {} 2021-11-20T08:56:59.741 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T08:56:59.742 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2021-11-20T08:56:59.742 INFO:tasks.cephfs.mount:self.client.name = client.0 2021-11-20T08:56:59.743 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2021-11-20T08:56:59.743 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T08:56:59.743 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T08:56:59.744 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T08:56:59.744 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2021-11-20T08:56:59.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:56:59.745 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2021-11-20T08:56:59.761 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T08:56:59.762 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T08:56:59.762 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T08:56:59.763 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2021-11-20T08:56:59.763 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2021-11-20T08:56:59.763 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2021-11-20T08:56:59.763 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T08:56:59.764 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2021-11-20T08:56:59.764 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T08:56:59.764 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2021-11-20T08:56:59.764 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T08:56:59.765 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2021-11-20T08:56:59.765 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T08:56:59.765 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2021-11-20T08:56:59.765 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-20T08:56:59.766 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42262sec preferred_lft 42262sec 2021-11-20T08:56:59.766 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2021-11-20T08:56:59.766 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2021-11-20T08:56:59.767 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T08:56:59.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:56:59.768 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2021-11-20T08:56:59.768 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2021-11-20T08:56:59.768 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link add name ceph-brx type bridge 2021-11-20T08:56:59.769 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip addr flush dev ceph-brx 2021-11-20T08:56:59.769 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set ceph-brx up 2021-11-20T08:56:59.769 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T08:56:59.770 DEBUG:teuthology.orchestra.run.smithi093:> ') 2021-11-20T08:56:59.838 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: audit 2021-11-20T08:56:58.862977+0000 mon.smithi093 (mon.0) 524 : 2021-11-20T08:56:59.838 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: audit [DBG] from='client.? 172.21.15.93:0/2885055880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T08:56:59.839 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: audit 2021-11-20T08:56:59.174509+0000 mon.smithi093 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.93:0/1177491017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T08:56:59.839 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: audit 2021-11-20T08 2021-11-20T08:56:59.840 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: :56:59.513389+0000 mon.smithi102 (mon.1) 16 : audit [DBG] from='client.? 172.21.15.93:0/2505128894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:56:59.840 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: cluster 2021-11-20T08:56: 2021-11-20T08:56:59.840 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: 59.524873+0000 mgr.smithi093.qdikkm (mgr.14162) 193 2021-11-20T08:56:59.841 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:56:59 smithi093 conmon[33522]: : cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2021-11-20T08:56:59.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:56:59.954 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T08:57:00.019 INFO:teuthology.orchestra.run.smithi093.stdout:1 2021-11-20T08:57:00.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:00.021 DEBUG:teuthology.orchestra.run.smithi093:> route 2021-11-20T08:57:00.104 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2021-11-20T08:57:00.104 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T08:57:00.104 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-20T08:57:00.104 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-20T08:57:00.105 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T08:57:00.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:00.106 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2021-11-20T08:57:00.106 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2021-11-20T08:57:00.106 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-20T08:57:00.107 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-20T08:57:00.107 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-20T08:57:00.107 DEBUG:teuthology.orchestra.run.smithi093:> ') 2021-11-20T08:57:00.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: audit 2021-11-20T08:56:58. 2021-11-20T08:57:00.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: 862977+0000 mon.smithi093 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.93:0/2885055880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T08:57:00.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: audit 2021-11-20T08:57:00.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: 2021-11-20T08:56:59.174509+0000 mon.smithi093 (mon.0) 525 : audit 2021-11-20T08:57:00.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: [DBG] from='client.? 172.21.15.93:0/1177491017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T08:57:00.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:57:00.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: 20T08:56:59.513389+0000 mon.smithi102 ( 2021-11-20T08:57:00.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: mon.1) 16 : audit [DBG] from='client.? 172.21.15.93:0/2505128894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:57:00.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:57:00.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: -11-20T08:56:59.524873+0000 2021-11-20T08:57:00.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 193 : cluster [DBG] 2021-11-20T08:57:00.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:56:59 smithi102 conmon[36472]: pgmap v156: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2021-11-20T08:57:00.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:00.469 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2021-11-20T08:57:00.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:00.487 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list-id 2021-11-20T08:57:00.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:00.543 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2021-11-20T08:57:00.543 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2021-11-20T08:57:00.543 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2021-11-20T08:57:00.544 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2021-11-20T08:57:00.544 DEBUG:teuthology.orchestra.run.smithi093:> ') 2021-11-20T08:57:00.634 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2021-11-20T08:57:00.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:00.635 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2021-11-20T08:57:00.636 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2021-11-20T08:57:00.636 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2021-11-20T08:57:00.636 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T08:57:00.637 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2021-11-20T08:57:00.637 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2021-11-20T08:57:00.637 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2021-11-20T08:57:00.638 DEBUG:teuthology.orchestra.run.smithi093:> ') 2021-11-20T08:57:00.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:00.871 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2021-11-20T08:57:00.871 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2021-11-20T08:57:00.872 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2021-11-20T08:57:00.872 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T08:57:00.872 DEBUG:teuthology.orchestra.run.smithi093:> ') 2021-11-20T08:57:00.978 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T08:57:01.120 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/20016822 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f5fe0 msgr2=0x7fbfbc0f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.120 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/20016822 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f5fe0 0x7fbfbc0f6440 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fbfb0004660 tx=0x7fbfb0009b30).stop 2021-11-20T08:57:01.120 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/20016822 shutdown_connections 2021-11-20T08:57:01.121 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/20016822 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f5fe0 0x7fbfbc0f6440 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.121 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/20016822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfbc0f4d40 0x7fbfbc0f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.122 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/20016822 >> 172.21.15.93:0/20016822 conn(0x7fbfbc0f04a0 msgr2=0x7fbfbc0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:01.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/20016822 shutdown_connections 2021-11-20T08:57:01.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/20016822 wait complete. 2021-11-20T08:57:01.123 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 Processor -- start 2021-11-20T08:57:01.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.116+0000 7fbfc1fe9700 1 -- start start 2021-11-20T08:57:01.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfc1fe9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f4d40 0x7fbfbc0f9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfc1fe9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfbc0f5fe0 0x7fbfbc0f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfc1fe9700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc0fa330 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfc1fe9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc0f8920 con 0x7fbfbc0f5fe0 2021-11-20T08:57:01.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbb7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f4d40 0x7fbfbc0f9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbb7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f4d40 0x7fbfbc0f9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43144/0 (socket says 172.21.15.93:43144) 2021-11-20T08:57:01.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbb7fe700 1 -- 172.21.15.93:0/3560863156 learned_addr learned my addr 172.21.15.93:0/3560863156 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:01.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbaffd700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfbc0f5fe0 0x7fbfbc0f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbb7fe700 1 -- 172.21.15.93:0/3560863156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfbc0f5fe0 msgr2=0x7fbfbc0f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbb7fe700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfbc0f5fe0 0x7fbfbc0f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbb7fe700 1 -- 172.21.15.93:0/3560863156 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb0005040 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbaffd700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfbc0f5fe0 0x7fbfbc0f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T08:57:01.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfbb7fe700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f4d40 0x7fbfbc0f9d80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fbfac002700 tx=0x7fbfac009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:01.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfa3fff700 1 -- 172.21.15.93:0/3560863156 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfac009320 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfbc0f8c00 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfa3fff700 1 -- 172.21.15.93:0/3560863156 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfac00f040 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.131 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfa3fff700 1 -- 172.21.15.93:0/3560863156 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfac014900 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.131 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.117+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfbc0f91c0 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.118+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfbc027a00 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.119+0000 7fbfa3fff700 1 -- 172.21.15.93:0/3560863156 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbfac009480 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.119+0000 7fbfa3fff700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfa405c2a0 0x7fbfa405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.119+0000 7fbfa3fff700 1 -- 172.21.15.93:0/3560863156 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fbfac076200 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.119+0000 7fbfbaffd700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfa405c2a0 0x7fbfa405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.119+0000 7fbfbaffd700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfa405c2a0 0x7fbfa405e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fbfb0009d60 tx=0x7fbfb0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:01.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.123+0000 7fbfa3fff700 1 -- 172.21.15.93:0/3560863156 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfac048ec0 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.278 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.279+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfbc0586e0 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.280+0000 7fbfa3fff700 1 -- 172.21.15.93:0/3560863156 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+14086 (secure 0 0 0) 0x7fbfac04c4e0 con 0x7fbfbc0f4d40 2021-11-20T08:57:01.279 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:57:01.280 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":43,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","created":"2021-11-20T08:51:57.464444+0000","modified":"2021-11-20T08:56:44.326907+0000","last_up_change":"2021-11-20T08:55:49.151657+0000","last_in_change":"2021-11-20T08:55:32.747237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T08:55:09.134409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T08:56:41.253240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T08:56:41.551098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b62eaab5-818e-4739-98c3-d386425d4837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6803","nonce":1548414389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6805","nonce":1548414389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6809","nonce":1548414389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6807","nonce":1548414389}]},"public_addr":"172.21.15.93:6803/1548414389","cluster_addr":"172.21.15.93:6805/1548414389","heartbeat_back_addr":"172.21.15.93:6809/1548414389","heartbeat_front_addr":"172.21.15.93:6807/1548414389","state":["exists","up"]},{"osd":1,"uuid":"a46f3336-3bc4-4736-9095-c4d72e662323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6811","nonce":3785015356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6813","nonce":3785015356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6817","nonce":3785015356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6815","nonce":3785015356}]},"public_addr":"172.21.15.93:6811/3785015356","cluster_addr":"172.21.15.93:6813/3785015356","heartbeat_back_addr":"172.21.15.93:6817/3785015356","heartbeat_front_addr":"172.21.15.93:6815/3785015356","state":["exists","up"]},{"osd":2,"uuid":"72096543-af61-429d-a8d1-a77726a1f3eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6819","nonce":82486730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6821","nonce":82486730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6825","nonce":82486730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6823","nonce":82486730}]},"public_addr":"172.21.15.93:6819/82486730","cluster_addr":"172.21.15.93:6821/82486730","heartbeat_back_addr":"172.21.15.93:6825/82486730","heartbeat_front_addr":"172.21.15.93:6823/82486730","state":["exists","up"]},{"osd":3,"uuid":"37ba4275-7023-4f74-ba1f-7ad2f864476f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6801","nonce":1093924688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6803","nonce":1093924688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6807","nonce":1093924688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6805","nonce":1093924688}]},"public_addr":"172.21.15.102:6801/1093924688","cluster_addr":"172.21.15.102:6803/1093924688","heartbeat_back_addr":"172.21.15.102:6807/1093924688","heartbeat_front_addr":"172.21.15.102:6805/1093924688","state":["exists","up"]},{"osd":4,"uuid":"c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6809","nonce":2700333748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6811","nonce":2700333748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6815","nonce":2700333748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6813","nonce":2700333748}]},"public_addr":"172.21.15.102:6809/2700333748","cluster_addr":"172.21.15.102:6811/2700333748","heartbeat_back_addr":"172.21.15.102:6815/2700333748","heartbeat_front_addr":"172.21.15.102:6813/2700333748","state":["exists","up"]},{"osd":5,"uuid":"fab60808-4ace-4f26-9c42-f72340134a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6817","nonce":4083093586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6819","nonce":4083093586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6823","nonce":4083093586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6821","nonce":4083093586}]},"public_addr":"172.21.15.102:6817/4083093586","cluster_addr":"172.21.15.102:6819/4083093586","heartbeat_back_addr":"172.21.15.102:6823/4083093586","heartbeat_front_addr":"172.21.15.102:6821/4083093586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:21.010676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:41.314338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:57.961372+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:13.733626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:30.250474+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:47.331557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:0/3689853676":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/3631043468":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/485709266":"2021-11-21T08:52:37.986811+0000","172.21.15.93:6800/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1880974768":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1518372611":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/2364704567":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/1580490298":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6801/1580490298":"2021-11-21T08:52:15.847551+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T08:57:01.282 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfa405c2a0 msgr2=0x7fbfa405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.282 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfa405c2a0 0x7fbfa405e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fbfb0009d60 tx=0x7fbfb0009b30).stop 2021-11-20T08:57:01.282 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f4d40 msgr2=0x7fbfbc0f9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.282 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f4d40 0x7fbfbc0f9d80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fbfac002700 tx=0x7fbfac009d80).stop 2021-11-20T08:57:01.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 shutdown_connections 2021-11-20T08:57:01.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbfa405c2a0 0x7fbfa405e760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfbc0f4d40 0x7fbfbc0f9d80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 --2- 172.21.15.93:0/3560863156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfbc0f5fe0 0x7fbfbc0f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.284 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 >> 172.21.15.93:0/3560863156 conn(0x7fbfbc0f04a0 msgr2=0x7fbfbc0fa520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:01.284 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 shutdown_connections 2021-11-20T08:57:01.284 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.282+0000 7fbfc1fe9700 1 -- 172.21.15.93:0/3560863156 wait complete. 2021-11-20T08:57:01.293 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T08:55:09.134409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T08:56:41.253240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T08:56:41.551098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T08:57:01.294 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T08:57:01.419 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 -- 172.21.15.93:0/2278067919 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f5570 msgr2=0x7fbf600f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/2278067919 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f5570 0x7fbf600f5990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fbf50004660 tx=0x7fbf50009b30).stop 2021-11-20T08:57:01.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 -- 172.21.15.93:0/2278067919 shutdown_connections 2021-11-20T08:57:01.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/2278067919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf600f6810 0x7fbf600f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/2278067919 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f5570 0x7fbf600f5990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 -- 172.21.15.93:0/2278067919 >> 172.21.15.93:0/2278067919 conn(0x7fbf600f0cf0 msgr2=0x7fbf600f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:01.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 -- 172.21.15.93:0/2278067919 shutdown_connections 2021-11-20T08:57:01.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.418+0000 7fbf64ba5700 1 -- 172.21.15.93:0/2278067919 wait complete. 2021-11-20T08:57:01.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf64ba5700 1 Processor -- start 2021-11-20T08:57:01.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf64ba5700 1 -- start start 2021-11-20T08:57:01.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf64ba5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf600f5570 0x7fbf60105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf64ba5700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f6810 0x7fbf60106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf64ba5700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60106890 con 0x7fbf600f6810 2021-11-20T08:57:01.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf64ba5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf601069d0 con 0x7fbf600f5570 2021-11-20T08:57:01.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf5dd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f6810 0x7fbf60106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf5dd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f6810 0x7fbf60106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43156/0 (socket says 172.21.15.93:43156) 2021-11-20T08:57:01.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf5dd9b700 1 -- 172.21.15.93:0/1350609592 learned_addr learned my addr 172.21.15.93:0/1350609592 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:01.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf5dd9b700 1 -- 172.21.15.93:0/1350609592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf600f5570 msgr2=0x7fbf60105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf5dd9b700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf600f5570 0x7fbf60105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:57:01.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.419+0000 7fbf5dd9b700 1 -- 172.21.15.93:0/1350609592 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50005040 con 0x7fbf600f6810 2021-11-20T08:57:01.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.420+0000 7fbf5dd9b700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f6810 0x7fbf60106290 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fbf5400c530 tx=0x7fbf54007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:01.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.420+0000 7fbf4effd700 1 -- 172.21.15.93:0/1350609592 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf5400f650 con 0x7fbf600f6810 2021-11-20T08:57:01.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.420+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6010b3e0 con 0x7fbf600f6810 2021-11-20T08:57:01.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.420+0000 7fbf4effd700 1 -- 172.21.15.93:0/1350609592 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf5400fc60 con 0x7fbf600f6810 2021-11-20T08:57:01.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.420+0000 7fbf4effd700 1 -- 172.21.15.93:0/1350609592 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf54012c80 con 0x7fbf600f6810 2021-11-20T08:57:01.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.420+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6005fe00 con 0x7fbf600f6810 2021-11-20T08:57:01.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.420+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf600fdd00 con 0x7fbf600f6810 2021-11-20T08:57:01.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.421+0000 7fbf4effd700 1 -- 172.21.15.93:0/1350609592 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbf54010070 con 0x7fbf600f6810 2021-11-20T08:57:01.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.421+0000 7fbf4effd700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbf4805c2a0 0x7fbf4805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.430 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.421+0000 7fbf4effd700 1 -- 172.21.15.93:0/1350609592 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fbf54075c00 con 0x7fbf600f6810 2021-11-20T08:57:01.430 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.421+0000 7fbf5e59c700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbf4805c2a0 0x7fbf4805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.430 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.422+0000 7fbf5e59c700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbf4805c2a0 0x7fbf4805e760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fbf50009f50 tx=0x7fbf50009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:01.431 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.425+0000 7fbf4effd700 1 -- 172.21.15.93:0/1350609592 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf540452b0 con 0x7fbf600f6810 2021-11-20T08:57:01.584 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.585+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fbf600ffc80 con 0x7fbf600f6810 2021-11-20T08:57:01.585 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.586+0000 7fbf4effd700 1 -- 172.21.15.93:0/1350609592 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v43) v1 ==== 110+0+10 (secure 0 0 0) 0x7fbf5407b020 con 0x7fbf600f6810 2021-11-20T08:57:01.585 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 1 2021-11-20T08:57:01.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbf4805c2a0 msgr2=0x7fbf4805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbf4805c2a0 0x7fbf4805e760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fbf50009f50 tx=0x7fbf50009b30).stop 2021-11-20T08:57:01.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f6810 msgr2=0x7fbf60106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f6810 0x7fbf60106290 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fbf5400c530 tx=0x7fbf54007e20).stop 2021-11-20T08:57:01.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 shutdown_connections 2021-11-20T08:57:01.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbf4805c2a0 0x7fbf4805e760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf600f5570 0x7fbf60105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:57:01.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 --2- 172.21.15.93:0/1350609592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf600f6810 0x7fbf60106290 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.587+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 >> 172.21.15.93:0/1350609592 conn(0x7fbf600f0cf0 msgr2=0x7fbf600fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:01.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.588+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 shutdown_connections 2021-11-20T08:57:01.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.588+0000 7fbf64ba5700 1 -- 172.21.15.93:0/1350609592 wait complete. 2021-11-20T08:57:01.599 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T08:57:01.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 -- 172.21.15.93:0/783481712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 msgr2=0x7f5b7c0f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 --2- 172.21.15.93:0/783481712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c0f5810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5b78004660 tx=0x7f5b78009b30).stop 2021-11-20T08:57:01.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 -- 172.21.15.93:0/783481712 shutdown_connections 2021-11-20T08:57:01.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 --2- 172.21.15.93:0/783481712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b7c0f6690 0x7f5b7c0f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 --2- 172.21.15.93:0/783481712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c0f5810 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 -- 172.21.15.93:0/783481712 >> 172.21.15.93:0/783481712 conn(0x7f5b7c0f0b30 msgr2=0x7f5b7c0f2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:01.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 -- 172.21.15.93:0/783481712 shutdown_connections 2021-11-20T08:57:01.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.726+0000 7f5b824d7700 1 -- 172.21.15.93:0/783481712 wait complete. 2021-11-20T08:57:01.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b824d7700 1 Processor -- start 2021-11-20T08:57:01.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b824d7700 1 -- start start 2021-11-20T08:57:01.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b824d7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b824d7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b7c0f6690 0x7f5b7c10bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b824d7700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0603b0 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b824d7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0604f0 con 0x7f5b7c0f6690 2021-11-20T08:57:01.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b814d5700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b814d5700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c109a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43164/0 (socket says 172.21.15.93:43164) 2021-11-20T08:57:01.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b814d5700 1 -- 172.21.15.93:0/253441063 learned_addr learned my addr 172.21.15.93:0/253441063 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:01.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b80cd4700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b7c0f6690 0x7f5b7c10bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b814d5700 1 -- 172.21.15.93:0/253441063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b7c0f6690 msgr2=0x7f5b7c10bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b814d5700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b7c0f6690 0x7f5b7c10bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b814d5700 1 -- 172.21.15.93:0/253441063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b78005040 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.738+0000 7f5b80cd4700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b7c0f6690 0x7f5b7c10bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T08:57:01.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.739+0000 7f5b814d5700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c109a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f5b78000c00 tx=0x7f5b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:01.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.739+0000 7f5b71ffb700 1 -- 172.21.15.93:0/253441063 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b78019070 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.739+0000 7f5b71ffb700 1 -- 172.21.15.93:0/253441063 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b78007e70 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.739+0000 7f5b71ffb700 1 -- 172.21.15.93:0/253441063 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b78014db0 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.743 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.739+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c10c4b0 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.743 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.739+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c10ca10 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.745 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.740+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c045ad0 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.745 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.740+0000 7f5b71ffb700 1 -- 172.21.15.93:0/253441063 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5b78025070 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.745 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.741+0000 7f5b71ffb700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5b6805c2b0 0x7f5b6805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:01.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.741+0000 7f5b71ffb700 1 -- 172.21.15.93:0/253441063 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f5b78076d30 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.741+0000 7f5b80cd4700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5b6805c2b0 0x7f5b6805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:01.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.741+0000 7f5b80cd4700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5b6805c2b0 0x7f5b6805e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5b6c009da0 tx=0x7f5b6c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:01.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.745+0000 7f5b71ffb700 1 -- 172.21.15.93:0/253441063 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b78049a70 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.907 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.908+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f5b7c0fdd00 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.908 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.909+0000 7f5b71ffb700 1 -- 172.21.15.93:0/253441063 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5b7804d090 con 0x7f5b7c0f53f0 2021-11-20T08:57:01.908 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 32 2021-11-20T08:57:01.910 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5b6805c2b0 msgr2=0x7f5b6805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.910 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5b6805c2b0 0x7f5b6805e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5b6c009da0 tx=0x7f5b6c00b040).stop 2021-11-20T08:57:01.910 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 msgr2=0x7f5b7c109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:01.911 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c109a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f5b78000c00 tx=0x7f5b7800b040).stop 2021-11-20T08:57:01.911 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 shutdown_connections 2021-11-20T08:57:01.911 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5b6805c2b0 0x7f5b6805e770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.911 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b7c0f53f0 0x7f5b7c109a60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.912 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 --2- 172.21.15.93:0/253441063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b7c0f6690 0x7f5b7c10bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:01.912 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 >> 172.21.15.93:0/253441063 conn(0x7f5b7c0f0b30 msgr2=0x7f5b7c0fa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:01.912 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 shutdown_connections 2021-11-20T08:57:01.912 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:01.910+0000 7f5b824d7700 1 -- 172.21.15.93:0/253441063 wait complete. 2021-11-20T08:57:01.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:01 smithi102 conmon[36472]: audit 2021-11-20T08:57:00.577269+0000 mon.smithi093 (mon.0) 526 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:57:01.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:01 smithi102 conmon[36472]: 2021-11-20T08:57:01.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:01 smithi102 conmon[36472]: audit 2021-11-20T08:57:01.280730+0000 mon.smithi093 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.93:0/3560863156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:57:01.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:01 smithi102 conmon[36472]: cluster 2021-11-20T08:57:01.525257+0000 mgr.smithi093.qdikkm (mgr.14162) 194 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-20T08:57:01.920 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:01 smithi093 conmon[33522]: audit 2021-11-20T08:57:00.577269+0000 mon.smithi093 (mon.0) 526 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:57:01.920 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:01 smithi093 conmon[33522]: audit 2021-11-20T08:57:01.280730+0000 mon.smithi093 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.93:0/3560863156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:57:01.920 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:01 smithi093 conmon[33522]: cluster 2021-11-20T08:57:01.525257+0000 mgr.smithi093.qdikkm (mgr.14162) 194 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-20T08:57:01.922 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T08:57:02.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.043+0000 7f05565b8700 1 -- 172.21.15.93:0/2225338564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f55a0 msgr2=0x7f05500f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.043+0000 7f05565b8700 1 --2- 172.21.15.93:0/2225338564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f55a0 0x7f05500f59c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f0538004660 tx=0x7f0538009b30).stop 2021-11-20T08:57:02.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.043+0000 7f05565b8700 1 -- 172.21.15.93:0/2225338564 shutdown_connections 2021-11-20T08:57:02.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.043+0000 7f05565b8700 1 --2- 172.21.15.93:0/2225338564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f6840 0x7f05500f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.043+0000 7f05565b8700 1 --2- 172.21.15.93:0/2225338564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f55a0 0x7f05500f59c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.043+0000 7f05565b8700 1 -- 172.21.15.93:0/2225338564 >> 172.21.15.93:0/2225338564 conn(0x7f05500f0d00 msgr2=0x7f05500f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 -- 172.21.15.93:0/2225338564 shutdown_connections 2021-11-20T08:57:02.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 -- 172.21.15.93:0/2225338564 wait complete. 2021-11-20T08:57:02.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 Processor -- start 2021-11-20T08:57:02.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 -- start start 2021-11-20T08:57:02.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f55a0 0x7f0550106e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f6840 0x7f0550107300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05501078b0 con 0x7f05500f55a0 2021-11-20T08:57:02.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.044+0000 7f05565b8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05501079f0 con 0x7f05500f6840 2021-11-20T08:57:02.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f55a0 0x7f0550106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f55a0 0x7f0550106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43174/0 (socket says 172.21.15.93:43174) 2021-11-20T08:57:02.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054ffff700 1 -- 172.21.15.93:0/2385039657 learned_addr learned my addr 172.21.15.93:0/2385039657 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:02.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054f7fe700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f6840 0x7f0550107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054ffff700 1 -- 172.21.15.93:0/2385039657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f6840 msgr2=0x7f0550107300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054ffff700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f6840 0x7f0550107300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054ffff700 1 -- 172.21.15.93:0/2385039657 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0538005040 con 0x7f05500f55a0 2021-11-20T08:57:02.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054ffff700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f55a0 0x7f0550106e10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0538000c00 tx=0x7f053800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054cff9700 1 -- 172.21.15.93:0/2385039657 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0538019070 con 0x7f05500f55a0 2021-11-20T08:57:02.050 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055010c430 con 0x7f05500f55a0 2021-11-20T08:57:02.050 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054cff9700 1 -- 172.21.15.93:0/2385039657 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0538007e70 con 0x7f05500f55a0 2021-11-20T08:57:02.050 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f054cff9700 1 -- 172.21.15.93:0/2385039657 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0538014d40 con 0x7f05500f55a0 2021-11-20T08:57:02.050 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.045+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055010c9f0 con 0x7f05500f55a0 2021-11-20T08:57:02.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.046+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05500fdd30 con 0x7f05500f55a0 2021-11-20T08:57:02.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.047+0000 7f054cff9700 1 -- 172.21.15.93:0/2385039657 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0538025070 con 0x7f05500f55a0 2021-11-20T08:57:02.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.047+0000 7f054cff9700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f053c05c2a0 0x7f053c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.047+0000 7f054cff9700 1 -- 172.21.15.93:0/2385039657 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f0538076e50 con 0x7f05500f55a0 2021-11-20T08:57:02.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.048+0000 7f054f7fe700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f053c05c2a0 0x7f053c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.048+0000 7f054f7fe700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f053c05c2a0 0x7f053c05e760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0540009fd0 tx=0x7f054000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.051+0000 7f054cff9700 1 -- 172.21.15.93:0/2385039657 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0538049b10 con 0x7f05500f55a0 2021-11-20T08:57:02.208 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.209+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f055010d030 con 0x7f05500f55a0 2021-11-20T08:57:02.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.209+0000 7f054cff9700 1 -- 172.21.15.93:0/2385039657 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7f053804d130 con 0x7f05500f55a0 2021-11-20T08:57:02.209 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 32 2021-11-20T08:57:02.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f053c05c2a0 msgr2=0x7f053c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f053c05c2a0 0x7f053c05e760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0540009fd0 tx=0x7f054000b040).stop 2021-11-20T08:57:02.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f55a0 msgr2=0x7f0550106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f55a0 0x7f0550106e10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0538000c00 tx=0x7f053800b040).stop 2021-11-20T08:57:02.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 shutdown_connections 2021-11-20T08:57:02.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f053c05c2a0 0x7f053c05e760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.213 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05500f55a0 0x7f0550106e10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.213 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 --2- 172.21.15.93:0/2385039657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05500f6840 0x7f0550107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.213 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 >> 172.21.15.93:0/2385039657 conn(0x7f05500f0d00 msgr2=0x7f0550060190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.213 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 shutdown_connections 2021-11-20T08:57:02.213 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.211+0000 7f05565b8700 1 -- 172.21.15.93:0/2385039657 wait complete. 2021-11-20T08:57:02.224 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T08:57:02.224 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:57:02.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.342+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/3735446346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 msgr2=0x7f9f380f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.342+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/3735446346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f380f7da0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f9f2c004660 tx=0x7f9f2c009b30).stop 2021-11-20T08:57:02.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.342+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/3735446346 shutdown_connections 2021-11-20T08:57:02.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.342+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/3735446346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f380f7da0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.342+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/3735446346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f380f66a0 0x7f9f380f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.342+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/3735446346 >> 172.21.15.93:0/3735446346 conn(0x7f9f380f1de0 msgr2=0x7f9f380f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/3735446346 shutdown_connections 2021-11-20T08:57:02.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/3735446346 wait complete. 2021-11-20T08:57:02.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 Processor -- start 2021-11-20T08:57:02.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 -- start start 2021-11-20T08:57:02.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f380f66a0 0x7f9f38106e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f38107320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f381078d0 con 0x7f9f380f7940 2021-11-20T08:57:02.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3cf2c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f38107a10 con 0x7f9f380f66a0 2021-11-20T08:57:02.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f35d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f38107320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f35d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f38107320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43186/0 (socket says 172.21.15.93:43186) 2021-11-20T08:57:02.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f35d9b700 1 -- 172.21.15.93:0/4008482594 learned_addr learned my addr 172.21.15.93:0/4008482594 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:02.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.343+0000 7f9f3659c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f380f66a0 0x7f9f38106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f35d9b700 1 -- 172.21.15.93:0/4008482594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f380f66a0 msgr2=0x7f9f38106e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f35d9b700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f380f66a0 0x7f9f38106e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f35d9b700 1 -- 172.21.15.93:0/4008482594 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c005040 con 0x7f9f380f7940 2021-11-20T08:57:02.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f3659c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f380f66a0 0x7f9f38106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T08:57:02.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f35d9b700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f38107320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f9f2c000c00 tx=0x7f9f2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f26ffd700 1 -- 172.21.15.93:0/4008482594 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f2c019070 con 0x7f9f380f7940 2021-11-20T08:57:02.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f26ffd700 1 -- 172.21.15.93:0/4008482594 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f2c007e70 con 0x7f9f380f7940 2021-11-20T08:57:02.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3810c450 con 0x7f9f380f7940 2021-11-20T08:57:02.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f26ffd700 1 -- 172.21.15.93:0/4008482594 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f2c014c90 con 0x7f9f380f7940 2021-11-20T08:57:02.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.344+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3810ca10 con 0x7f9f380f7940 2021-11-20T08:57:02.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.345+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f38027a00 con 0x7f9f380f7940 2021-11-20T08:57:02.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.346+0000 7f9f26ffd700 1 -- 172.21.15.93:0/4008482594 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9f2c025070 con 0x7f9f380f7940 2021-11-20T08:57:02.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.346+0000 7f9f26ffd700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9f2005c2a0 0x7f9f2005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.346+0000 7f9f26ffd700 1 -- 172.21.15.93:0/4008482594 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9f2c076e10 con 0x7f9f380f7940 2021-11-20T08:57:02.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.346+0000 7f9f3659c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9f2005c2a0 0x7f9f2005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.346+0000 7f9f3659c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9f2005c2a0 0x7f9f2005e760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9f28009fd0 tx=0x7f9f2800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.350+0000 7f9f26ffd700 1 -- 172.21.15.93:0/4008482594 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f2c049b50 con 0x7f9f380f7940 2021-11-20T08:57:02.544 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.545+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f380fee30 con 0x7f9f380f7940 2021-11-20T08:57:02.544 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.545+0000 7f9f26ffd700 1 -- 172.21.15.93:0/4008482594 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2932 (secure 0 0 0) 0x7f9f2c04d170 con 0x7f9f380f7940 2021-11-20T08:57:02.545 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:57:02.545 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14428,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6827/2783208877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":2783208877},{"type":"v1","addr":"172.21.15.93:6827","nonce":2783208877}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":10},{"gid":14438,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/1976124814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":1976124814},{"type":"v1","addr":"172.21.15.93:6829","nonce":1976124814}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":12},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:56:50.485290+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.102:6825/1423063558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1423063558},{"type":"v1","addr":"172.21.15.102:6825","nonce":1423063558}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:57:02.547 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9f2005c2a0 msgr2=0x7f9f2005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.547 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9f2005c2a0 0x7f9f2005e760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9f28009fd0 tx=0x7f9f2800b040).stop 2021-11-20T08:57:02.547 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 msgr2=0x7f9f38107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.547 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f38107320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f9f2c000c00 tx=0x7f9f2c00b040).stop 2021-11-20T08:57:02.548 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 shutdown_connections 2021-11-20T08:57:02.548 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9f2005c2a0 0x7f9f2005e760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.548 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f380f66a0 0x7f9f38106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.548 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 --2- 172.21.15.93:0/4008482594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f380f7940 0x7f9f38107320 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.548 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 >> 172.21.15.93:0/4008482594 conn(0x7f9f380f1de0 msgr2=0x7f9f380fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.549 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 shutdown_connections 2021-11-20T08:57:02.549 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.547+0000 7f9f3cf2c700 1 -- 172.21.15.93:0/4008482594 wait complete. 2021-11-20T08:57:02.549 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 12 2021-11-20T08:57:02.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:56:50.485290+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24257': {'gid': 24257, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.102:6825/1423063558', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1423063558}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1423063558}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:57:02.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-11-20T08:57:02.558 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi102.lhrlmh status 2021-11-20T08:57:02.693 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 -- 172.21.15.93:0/954349768 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f6690 msgr2=0x7fe3940f6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 --2- 172.21.15.93:0/954349768 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f6690 0x7fe3940f6ab0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fe390004660 tx=0x7fe390009b30).stop 2021-11-20T08:57:02.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 -- 172.21.15.93:0/954349768 shutdown_connections 2021-11-20T08:57:02.694 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 --2- 172.21.15.93:0/954349768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3940f7930 0x7fe3940f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 --2- 172.21.15.93:0/954349768 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f6690 0x7fe3940f6ab0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 -- 172.21.15.93:0/954349768 >> 172.21.15.93:0/954349768 conn(0x7fe3940f1e10 msgr2=0x7fe3940f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 -- 172.21.15.93:0/954349768 shutdown_connections 2021-11-20T08:57:02.695 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.681+0000 7fe39b71d700 1 -- 172.21.15.93:0/954349768 wait complete. 2021-11-20T08:57:02.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe39b71d700 1 Processor -- start 2021-11-20T08:57:02.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe39b71d700 1 -- start start 2021-11-20T08:57:02.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe39b71d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3940f6690 0x7fe394106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe39b71d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f7930 0x7fe394107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.696 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe39b71d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394107900 con 0x7fe3940f7930 2021-11-20T08:57:02.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe39b71d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394107a40 con 0x7fe3940f6690 2021-11-20T08:57:02.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe398cb8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f7930 0x7fe394107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.697 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe398cb8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f7930 0x7fe394107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43196/0 (socket says 172.21.15.93:43196) 2021-11-20T08:57:02.698 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe398cb8700 1 -- 172.21.15.93:0/3012411204 learned_addr learned my addr 172.21.15.93:0/3012411204 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:02.698 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe3994b9700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3940f6690 0x7fe394106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.698 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe398cb8700 1 -- 172.21.15.93:0/3012411204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3940f6690 msgr2=0x7fe394106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.699 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe398cb8700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3940f6690 0x7fe394106e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.699 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.682+0000 7fe398cb8700 1 -- 172.21.15.93:0/3012411204 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe390005040 con 0x7fe3940f7930 2021-11-20T08:57:02.699 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.683+0000 7fe398cb8700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f7930 0x7fe394107350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe3840026d0 tx=0x7fe384009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.699 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.683+0000 7fe389ffb700 1 -- 172.21.15.93:0/3012411204 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3840079c0 con 0x7fe3940f7930 2021-11-20T08:57:02.700 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.683+0000 7fe389ffb700 1 -- 172.21.15.93:0/3012411204 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe38400f040 con 0x7fe3940f7930 2021-11-20T08:57:02.700 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.683+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39410c4e0 con 0x7fe3940f7930 2021-11-20T08:57:02.700 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.683+0000 7fe389ffb700 1 -- 172.21.15.93:0/3012411204 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3840148b0 con 0x7fe3940f7930 2021-11-20T08:57:02.700 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.683+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39410caa0 con 0x7fe3940f7930 2021-11-20T08:57:02.701 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.684+0000 7fe389ffb700 1 -- 172.21.15.93:0/3012411204 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe384007b20 con 0x7fe3940f7930 2021-11-20T08:57:02.701 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.685+0000 7fe389ffb700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe38005c2a0 0x7fe38005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.701 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.685+0000 7fe389ffb700 1 -- 172.21.15.93:0/3012411204 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fe384075de0 con 0x7fe3940f7930 2021-11-20T08:57:02.702 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.685+0000 7fe3994b9700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe38005c2a0 0x7fe38005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.702 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.685+0000 7fe3994b9700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe38005c2a0 0x7fe38005e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe390009f50 tx=0x7fe390009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.702 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.686+0000 7fe3ad2c8b80 1 Processor -- start 2021-11-20T08:57:02.702 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.686+0000 7fe3ad2c8b80 1 -- start start 2021-11-20T08:57:02.703 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8710f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.703 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8712150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.703 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a864b4d0 con 0x5583a857a1d0 2021-11-20T08:57:02.704 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a865f3e0 con 0x5583a87113f0 2021-11-20T08:57:02.704 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3994b9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8710f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.704 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3994b9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8710f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43200/0 (socket says 172.21.15.93:43200) 2021-11-20T08:57:02.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3994b9700 1 -- 172.21.15.93:0/2616844380 learned_addr learned my addr 172.21.15.93:0/2616844380 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:02.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe398cb8700 1 --2- 172.21.15.93:0/2616844380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8712150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3994b9700 1 -- 172.21.15.93:0/2616844380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 msgr2=0x5583a8712150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T08:57:02.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3994b9700 1 --2- 172.21.15.93:0/2616844380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8712150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3994b9700 1 -- 172.21.15.93:0/2616844380 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583a86db3c0 con 0x5583a857a1d0 2021-11-20T08:57:02.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe3994b9700 1 --2- 172.21.15.93:0/2616844380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8710f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fe3940f7790 tx=0x7fe39001d000).ready entity=mon.0 client_cookie=1e2571ce600ff1c4 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe37effd700 1 -- 172.21.15.93:0/2616844380 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39000e040 con 0x5583a857a1d0 2021-11-20T08:57:02.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe37effd700 1 -- 172.21.15.93:0/2616844380 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe390019040 con 0x5583a857a1d0 2021-11-20T08:57:02.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.687+0000 7fe37effd700 1 -- 172.21.15.93:0/2616844380 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39001da90 con 0x5583a857a1d0 2021-11-20T08:57:02.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2616844380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 msgr2=0x5583a8710f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2616844380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8710f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fe3940f7790 tx=0x7fe39001d000).stop 2021-11-20T08:57:02.708 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2616844380 shutdown_connections 2021-11-20T08:57:02.708 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2616844380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8712150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2616844380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8710f00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2616844380 >> 172.21.15.93:0/2616844380 conn(0x5583a86d29b0 msgr2=0x5583a8707fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2616844380 shutdown_connections 2021-11-20T08:57:02.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2616844380 wait complete. 2021-11-20T08:57:02.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 -- start start 2021-11-20T08:57:02.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.688+0000 7fe3ad2c8b80 1 Processor -- start 2021-11-20T08:57:02.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a857a1d0 0x5583a8719b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a87113f0 0x5583a8719ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a865f520 con 0x5583a87113f0 2021-11-20T08:57:02.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a865faa0 con 0x5583a857a1d0 2021-11-20T08:57:02.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 10 client.0 ll_register_callbacks cb 0x5583a86cc700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T08:57:02.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe398cb8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a87113f0 0x5583a8719ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe398cb8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a87113f0 0x5583a8719ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43206/0 (socket says 172.21.15.93:43206) 2021-11-20T08:57:02.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe398cb8700 1 -- 172.21.15.93:0/2900241161 learned_addr learned my addr 172.21.15.93:0/2900241161 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:02.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3994b9700 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a857a1d0 0x5583a8719b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe398cb8700 1 -- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a857a1d0 msgr2=0x5583a8719b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T08:57:02.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe398cb8700 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a857a1d0 0x5583a8719b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe398cb8700 1 -- 172.21.15.93:0/2900241161 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3940f8410 con 0x5583a87113f0 2021-11-20T08:57:02.714 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe398cb8700 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a87113f0 0x5583a8719ff0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fe384002d70 tx=0x7fe38404e040).ready entity=mon.0 client_cookie=43bad7ecb081bbef server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.714 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe373fff700 10 client.0 ms_handle_connect on v2:172.21.15.93:3300/0 2021-11-20T08:57:02.714 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe373fff700 1 -- 172.21.15.93:0/2900241161 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe38404ca10 con 0x5583a87113f0 2021-11-20T08:57:02.715 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe373fff700 1 -- 172.21.15.93:0/2900241161 <== mon.0 v2:172.21.15.93:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fe38404cb70 con 0x5583a87113f0 2021-11-20T08:57:02.715 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583a86dace0 con 0x5583a87113f0 2021-11-20T08:57:02.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe373fff700 1 -- 172.21.15.93:0/2900241161 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe38404cd30 con 0x5583a87113f0 2021-11-20T08:57:02.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 10 client.14498 fetch_fsmap learned FSMap version 12 2021-11-20T08:57:02.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.689+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x5583a864e630 con 0x5583a87113f0 2021-11-20T08:57:02.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe373fff700 1 -- 172.21.15.93:0/2900241161 <== mon.0 v2:172.21.15.93:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x5583a86dace0 con 0x5583a87113f0 2021-11-20T08:57:02.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe373fff700 1 -- 172.21.15.93:0/2900241161 <== mon.0 v2:172.21.15.93:3300/0 5 ==== fsmap(e 12) v1 ==== 1724+0+0 (secure 0 0 0) 0x7fe38401b020 con 0x5583a87113f0 2021-11-20T08:57:02.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3ad2c8b80 10 client.14498 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T08:57:02.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3ad2c8b80 10 client.14498 resolve_mds: resolved name 'cephfs.smithi102.lhrlmh' to daemon mds.cephfs.smithi102.lhrlmh 2021-11-20T08:57:02.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3ad2c8b80 20 client.14498 populate_metadata read hostname 'smithi093' 2021-11-20T08:57:02.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8722900 0x5583a8724cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3ad2c8b80 4 client.14498 mds_command: new command op to 24257 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T08:57:02.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 --> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583a8655df0 con 0x5583a8722900 2021-11-20T08:57:02.721 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3994b9700 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8722900 0x5583a8724cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe373fff700 10 client.14498 ms_handle_connect on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:02.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.690+0000 7fe3994b9700 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8722900 0x5583a8724cf0 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=26c94c30382d40f5 server_cookie=c356edbaf2cac1ea in_seq=0 out_seq=0 2021-11-20T08:57:02.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe373fff700 1 -- 172.21.15.93:0/2900241161 <== mds.0 v2:172.21.15.102:6824/1423063558 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x5583a8655df0 con 0x5583a8722900 2021-11-20T08:57:02.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe373fff700 10 client.14498 handle_command_reply: tid=0 2021-11-20T08:57:02.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 client.14498 shutdown 2021-11-20T08:57:02.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a87113f0 msgr2=0x5583a8719ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a87113f0 0x5583a8719ff0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fe384002d70 tx=0x7fe38404e040).stop 2021-11-20T08:57:02.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 shutdown_connections 2021-11-20T08:57:02.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8722900 0x5583a8724cf0 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a857a1d0 0x5583a8719b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/2900241161 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a87113f0 0x5583a8719ff0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe373fff700 0 client.14498 ms_handle_reset on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:02.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 >> 172.21.15.93:0/2900241161 conn(0x5583a86d29b0 msgr2=0x5583a87091d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 shutdown_connections 2021-11-20T08:57:02.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/2900241161 wait complete. 2021-11-20T08:57:02.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.691+0000 7fe3ad2c8b80 20 client.14498 trim_cache size 0 max 16384 2021-11-20T08:57:02.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3ad2c8b80 1 Processor -- start 2021-11-20T08:57:02.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3ad2c8b80 1 -- start start 2021-11-20T08:57:02.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8718830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8718d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a862d4a0 con 0x5583a857a1d0 2021-11-20T08:57:02.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a8655df0 con 0x5583a87113f0 2021-11-20T08:57:02.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3994b9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8718830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3994b9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8718830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43210/0 (socket says 172.21.15.93:43210) 2021-11-20T08:57:02.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe3994b9700 1 -- 172.21.15.93:0/344718814 learned_addr learned my addr 172.21.15.93:0/344718814 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:02.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.707+0000 7fe398cb8700 1 --2- 172.21.15.93:0/344718814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8718d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3994b9700 1 -- 172.21.15.93:0/344718814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 msgr2=0x5583a8718d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T08:57:02.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3994b9700 1 --2- 172.21.15.93:0/344718814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8718d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3994b9700 1 -- 172.21.15.93:0/344718814 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583a8705490 con 0x5583a857a1d0 2021-11-20T08:57:02.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3994b9700 1 --2- 172.21.15.93:0/344718814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8718830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fe3940f7790 tx=0x7fe39002b040).ready entity=mon.0 client_cookie=25c95cac76a1b96c server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe37f7fe700 1 -- 172.21.15.93:0/344718814 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39002e040 con 0x5583a857a1d0 2021-11-20T08:57:02.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe37f7fe700 1 -- 172.21.15.93:0/344718814 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe390019040 con 0x5583a857a1d0 2021-11-20T08:57:02.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe37f7fe700 1 -- 172.21.15.93:0/344718814 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39000e040 con 0x5583a857a1d0 2021-11-20T08:57:02.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/344718814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 msgr2=0x5583a8718830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/344718814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8718830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fe3940f7790 tx=0x7fe39002b040).stop 2021-11-20T08:57:02.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/344718814 shutdown_connections 2021-11-20T08:57:02.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/344718814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a8718d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/344718814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a8718830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/344718814 >> 172.21.15.93:0/344718814 conn(0x5583a86d29b0 msgr2=0x5583a87098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/344718814 shutdown_connections 2021-11-20T08:57:02.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.708+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/344718814 wait complete. 2021-11-20T08:57:02.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3ad2c8b80 1 -- start start 2021-11-20T08:57:02.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3ad2c8b80 1 Processor -- start 2021-11-20T08:57:02.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a876e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3ad2c8b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a876ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a876f3f0 con 0x5583a857a1d0 2021-11-20T08:57:02.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3ad2c8b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583a876f530 con 0x5583a87113f0 2021-11-20T08:57:02.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3ad2c8b80 10 client.0 ll_register_callbacks cb 0x5583a86cc700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T08:57:02.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3994b9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a876e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.741 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3994b9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a876e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43214/0 (socket says 172.21.15.93:43214) 2021-11-20T08:57:02.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3994b9700 1 -- 172.21.15.93:0/4215739132 learned_addr learned my addr 172.21.15.93:0/4215739132 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:02.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe398cb8700 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a876ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3994b9700 1 -- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 msgr2=0x5583a876ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T08:57:02.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3994b9700 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a876ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.742 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3994b9700 1 -- 172.21.15.93:0/4215739132 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583a86db3c0 con 0x5583a857a1d0 2021-11-20T08:57:02.743 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe3994b9700 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a876e950 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fe390009e10 tx=0x7fe39002b040).ready entity=mon.0 client_cookie=2c9338d738a5242b server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:02.743 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.709+0000 7fe373fff700 10 client.0 ms_handle_connect on v2:172.21.15.93:3300/0 2021-11-20T08:57:02.743 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe373fff700 1 -- 172.21.15.93:0/4215739132 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39002e070 con 0x5583a857a1d0 2021-11-20T08:57:02.743 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe373fff700 1 -- 172.21.15.93:0/4215739132 <== mon.0 v2:172.21.15.93:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fe390007c40 con 0x5583a857a1d0 2021-11-20T08:57:02.744 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583a8707220 con 0x5583a857a1d0 2021-11-20T08:57:02.744 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe373fff700 1 -- 172.21.15.93:0/4215739132 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe39000e040 con 0x5583a857a1d0 2021-11-20T08:57:02.744 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 10 client.14502 fetch_fsmap learned FSMap version 12 2021-11-20T08:57:02.745 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x7fe39406b690 con 0x5583a857a1d0 2021-11-20T08:57:02.745 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe373fff700 1 -- 172.21.15.93:0/4215739132 <== mon.0 v2:172.21.15.93:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x5583a8707220 con 0x5583a857a1d0 2021-11-20T08:57:02.745 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe373fff700 1 -- 172.21.15.93:0/4215739132 <== mon.0 v2:172.21.15.93:3300/0 5 ==== fsmap(e 12) v1 ==== 1724+0+0 (secure 0 0 0) 0x7fe39003b3f0 con 0x5583a857a1d0 2021-11-20T08:57:02.745 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 10 client.14502 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T08:57:02.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 10 client.14502 resolve_mds: resolved name 'cephfs.smithi102.lhrlmh' to daemon mds.cephfs.smithi102.lhrlmh 2021-11-20T08:57:02.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 20 client.14502 populate_metadata read hostname 'smithi093' 2021-11-20T08:57:02.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8774c20 0x5583a8777010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:02.746 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 4 client.14502 mds_command: new command op to 24257 tid=0[{"prefix": "status"}] 2021-11-20T08:57:02.747 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 --> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5583a864d0c0 con 0x5583a8774c20 2021-11-20T08:57:02.747 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.710+0000 7fe398cb8700 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8774c20 0x5583a8777010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:02.747 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe373fff700 10 client.14502 ms_handle_connect on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:02.747 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe398cb8700 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8774c20 0x5583a8777010 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=3a454b14cd743224 server_cookie=c446081a926740fa in_seq=0 out_seq=0 2021-11-20T08:57:02.747 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe373fff700 1 -- 172.21.15.93:0/4215739132 <== mds.0 v2:172.21.15.102:6824/1423063558 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5583a864d0c0 con 0x5583a8774c20 2021-11-20T08:57:02.748 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe373fff700 10 client.14502 handle_command_reply: tid=0 2021-11-20T08:57:02.748 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 client.14502 shutdown 2021-11-20T08:57:02.748 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 msgr2=0x5583a876e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.748 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a876e950 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fe390009e10 tx=0x7fe39002b040).stop 2021-11-20T08:57:02.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 shutdown_connections 2021-11-20T08:57:02.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x5583a8774c20 0x5583a8777010 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5583a857a1d0 0x5583a876e950 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe373fff700 0 client.14502 ms_handle_reset on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:02.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 --2- 172.21.15.93:0/4215739132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5583a87113f0 0x5583a876ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 >> 172.21.15.93:0/4215739132 conn(0x5583a86d29b0 msgr2=0x5583a8718830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 shutdown_connections 2021-11-20T08:57:02.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 1 -- 172.21.15.93:0/4215739132 wait complete. 2021-11-20T08:57:02.751 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.711+0000 7fe3ad2c8b80 20 client.14502 trim_cache size 0 max 16384 2021-11-20T08:57:02.751 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2021-11-20T08:57:02.751 INFO:teuthology.orchestra.run.smithi093.stdout: "cluster_fsid": "11458b34-49df-11ec-8c2c-001a4aab830c", 2021-11-20T08:57:02.752 INFO:teuthology.orchestra.run.smithi093.stdout: "whoami": 0, 2021-11-20T08:57:02.752 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 24257, 2021-11-20T08:57:02.752 INFO:teuthology.orchestra.run.smithi093.stdout: "want_state": "up:active", 2021-11-20T08:57:02.752 INFO:teuthology.orchestra.run.smithi093.stdout: "state": "up:active", 2021-11-20T08:57:02.753 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs", 2021-11-20T08:57:02.753 INFO:teuthology.orchestra.run.smithi093.stdout: "rank_uptime": 17.368571943999999, 2021-11-20T08:57:02.753 INFO:teuthology.orchestra.run.smithi093.stdout: "mdsmap_epoch": 9, 2021-11-20T08:57:02.753 INFO:teuthology.orchestra.run.smithi093.stdout: "osdmap_epoch": 43, 2021-11-20T08:57:02.754 INFO:teuthology.orchestra.run.smithi093.stdout: "osdmap_epoch_barrier": 43, 2021-11-20T08:57:02.754 INFO:teuthology.orchestra.run.smithi093.stdout: "uptime": 17.901094127 2021-11-20T08:57:02.754 INFO:teuthology.orchestra.run.smithi093.stdout:} 2021-11-20T08:57:02.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe38005c2a0 msgr2=0x7fe38005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe38005c2a0 0x7fe38005e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe390009f50 tx=0x7fe390009b30).stop 2021-11-20T08:57:02.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f7930 msgr2=0x7fe394107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:02.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f7930 0x7fe394107350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe3840026d0 tx=0x7fe384009de0).stop 2021-11-20T08:57:02.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 shutdown_connections 2021-11-20T08:57:02.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe38005c2a0 0x7fe38005e760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3940f6690 0x7fe394106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 --2- 172.21.15.93:0/3012411204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe3940f7930 0x7fe394107350 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:02.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 >> 172.21.15.93:0/3012411204 conn(0x7fe3940f1e10 msgr2=0x7fe3940fb380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:02.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 shutdown_connections 2021-11-20T08:57:02.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:02.712+0000 7fe39b71d700 1 -- 172.21.15.93:0/3012411204 wait complete. 2021-11-20T08:57:02.759 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2021-11-20T08:57:02.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:02.759 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p /home/ubuntu/cephtest/mnt.0 2021-11-20T08:57:02.776 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-11-20T08:57:02.776 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T08:57:02.837 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:02 smithi093 conmon[33522]: audit 2021-11-20T08:57:01.586629+0000 mon.smithi093 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.93:0/1350609592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:57:02.838 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:02 smithi093 conmon[33522]: audit 2021-11-20T08:57:01.909828+0000 mon.smithi093 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.93:0/253441063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T08:57:02.838 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:02 smithi093 conmon[33522]: audit 2021-11-20T08:57:02.210265+0000 mon.smithi093 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.93:0/2385039657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T08:57:02.838 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:02 smithi093 conmon[33522]: audit 2021-11-20T08:57:02.546478+0000 mon.smithi093 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.93:0/4008482594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:57:02.868 INFO:teuthology.orchestra.run.smithi093.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T08:57:02.869 INFO:teuthology.orchestra.run.smithi093.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T08:57:02.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:02 smithi102 conmon[36472]: audit 2021-11-20T08:57:01.586629+0000 mon.smithi093 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.93:0/1350609592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:57:02.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:02 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:57:02.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:02 smithi102 conmon[36472]: T08:57:01.909828+0000 mon.smithi093 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.93:0/253441063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T08:57:02.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:02 smithi102 conmon[36472]: audit 2021-11-20T08:57:02.210265+0000 mon.smithi093 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.93:0/2385039657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T08:57:02.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:02 smithi102 conmon[36472]: audit 2021-11-20T08:57:02.546478+0000 mon.smithi093 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.93:0/4008482594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:57:02.993 INFO:teuthology.orchestra.run.smithi093.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T08:57:02.993 INFO:teuthology.orchestra.run.smithi093.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T08:57:02.997 INFO:tasks.cephfs.kernel_mount:mount command passed 2021-11-20T08:57:02.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:02.998 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2021-11-20T08:57:03.064 INFO:tasks.kclient:client.1 config is {} 2021-11-20T08:57:03.065 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T08:57:03.065 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2021-11-20T08:57:03.066 INFO:tasks.cephfs.mount:self.client.name = client.1 2021-11-20T08:57:03.067 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2021-11-20T08:57:03.068 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T08:57:03.068 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T08:57:03.069 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T08:57:03.069 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2021-11-20T08:57:03.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.070 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2021-11-20T08:57:03.092 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T08:57:03.093 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T08:57:03.093 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T08:57:03.094 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2021-11-20T08:57:03.094 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2021-11-20T08:57:03.094 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2021-11-20T08:57:03.095 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T08:57:03.095 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2021-11-20T08:57:03.095 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T08:57:03.096 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2021-11-20T08:57:03.096 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T08:57:03.097 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2021-11-20T08:57:03.097 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T08:57:03.097 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2021-11-20T08:57:03.098 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-20T08:57:03.098 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42256sec preferred_lft 42256sec 2021-11-20T08:57:03.098 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link noprefixroute 2021-11-20T08:57:03.099 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2021-11-20T08:57:03.099 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T08:57:03.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.100 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2021-11-20T08:57:03.100 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2021-11-20T08:57:03.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2021-11-20T08:57:03.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2021-11-20T08:57:03.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2021-11-20T08:57:03.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T08:57:03.102 DEBUG:teuthology.orchestra.run.smithi102:> ') 2021-11-20T08:57:03.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.272 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T08:57:03.298 INFO:teuthology.orchestra.run.smithi102.stdout:1 2021-11-20T08:57:03.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.301 DEBUG:teuthology.orchestra.run.smithi102:> route 2021-11-20T08:57:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2021-11-20T08:57:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T08:57:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-20T08:57:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-20T08:57:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T08:57:03.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.373 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2021-11-20T08:57:03.373 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2021-11-20T08:57:03.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-20T08:57:03.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-20T08:57:03.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-20T08:57:03.375 DEBUG:teuthology.orchestra.run.smithi102:> ') 2021-11-20T08:57:03.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.604 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2021-11-20T08:57:03.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.620 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2021-11-20T08:57:03.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.675 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2021-11-20T08:57:03.676 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2021-11-20T08:57:03.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2021-11-20T08:57:03.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2021-11-20T08:57:03.677 DEBUG:teuthology.orchestra.run.smithi102:> ') 2021-11-20T08:57:03.767 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2021-11-20T08:57:03.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.768 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2021-11-20T08:57:03.769 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2021-11-20T08:57:03.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2021-11-20T08:57:03.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T08:57:03.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2021-11-20T08:57:03.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2021-11-20T08:57:03.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2021-11-20T08:57:03.773 DEBUG:teuthology.orchestra.run.smithi102:> ') 2021-11-20T08:57:03.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T08:57:03.980 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2021-11-20T08:57:03.980 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2021-11-20T08:57:03.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2021-11-20T08:57:03.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T08:57:03.981 DEBUG:teuthology.orchestra.run.smithi102:> ') 2021-11-20T08:57:04.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:03 smithi102 conmon[36472]: audit 2021-11-20T08:57:02.779502 2021-11-20T08:57:04.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:03 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 532 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:57:04.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:03 smithi102 conmon[36472]: audit 2021-11-20T08:57:02.785087+0000 mon.smithi093 (mon.0) 533 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:57:04.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:03 smithi102 conmon[36472]: cluster 2021-11-20T08:57:03.525789+0000 mgr.smithi093.qdikkm (mgr.14162) 195 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-20T08:57:04.089 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T08:57:04.117 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:03 smithi093 conmon[33522]: audit 2021-11-20T08:57:02.779502+0000 mon.smithi093 (mon. 2021-11-20T08:57:04.117 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:03 smithi093 conmon[33522]: 0) 532 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:57:04.118 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:03 smithi093 conmon[33522]: audit 2021-11-20T08:57:02.785087+0000 mon.smithi093 (mon.0) 533 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:57:04.118 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:03 smithi093 conmon[33522]: cluster 2021-11-20T08:57:03.525789+0000 mgr.smithi093.qdikkm (mgr.14162) 195 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-20T08:57:04.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 -- 172.21.15.93:0/1313226623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f6840 msgr2=0x7f047c0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 --2- 172.21.15.93:0/1313226623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f6840 0x7f047c0f6ca0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f0470004660 tx=0x7f0470009b30).stop 2021-11-20T08:57:04.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 -- 172.21.15.93:0/1313226623 shutdown_connections 2021-11-20T08:57:04.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 --2- 172.21.15.93:0/1313226623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f6840 0x7f047c0f6ca0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 --2- 172.21.15.93:0/1313226623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f047c0f55a0 0x7f047c0f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 -- 172.21.15.93:0/1313226623 >> 172.21.15.93:0/1313226623 conn(0x7f047c0f0ce0 msgr2=0x7f047c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:04.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 -- 172.21.15.93:0/1313226623 shutdown_connections 2021-11-20T08:57:04.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.218+0000 7f04822b7700 1 -- 172.21.15.93:0/1313226623 wait complete. 2021-11-20T08:57:04.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f04822b7700 1 Processor -- start 2021-11-20T08:57:04.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f04822b7700 1 -- start start 2021-11-20T08:57:04.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f04822b7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f55a0 0x7f047c0f9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f04822b7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f047c0f6840 0x7f047c0f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f04822b7700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0f9ad0 con 0x7f047c0f55a0 2021-11-20T08:57:04.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f04822b7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0f8070 con 0x7f047c0f6840 2021-11-20T08:57:04.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f047bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f55a0 0x7f047c0f9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f047bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f55a0 0x7f047c0f9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43228/0 (socket says 172.21.15.93:43228) 2021-11-20T08:57:04.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f047bfff700 1 -- 172.21.15.93:0/162778818 learned_addr learned my addr 172.21.15.93:0/162778818 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:04.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.219+0000 7f047b7fe700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f047c0f6840 0x7f047c0f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f047bfff700 1 -- 172.21.15.93:0/162778818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f047c0f6840 msgr2=0x7f047c0f7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f047bfff700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f047c0f6840 0x7f047c0f7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f047bfff700 1 -- 172.21.15.93:0/162778818 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0470005040 con 0x7f047c0f55a0 2021-11-20T08:57:04.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f047bfff700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f55a0 0x7f047c0f9520 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f046c00e7c0 tx=0x7f046c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:04.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f0478ff9700 1 -- 172.21.15.93:0/162778818 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f046c0092e0 con 0x7f047c0f55a0 2021-11-20T08:57:04.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f0478ff9700 1 -- 172.21.15.93:0/162778818 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f046c00f040 con 0x7f047c0f55a0 2021-11-20T08:57:04.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047c0f82f0 con 0x7f047c0f55a0 2021-11-20T08:57:04.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f0478ff9700 1 -- 172.21.15.93:0/162778818 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f046c0147c0 con 0x7f047c0f55a0 2021-11-20T08:57:04.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.220+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047c0f88b0 con 0x7f047c0f55a0 2021-11-20T08:57:04.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.222+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047c0fdd30 con 0x7f047c0f55a0 2021-11-20T08:57:04.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.222+0000 7f0478ff9700 1 -- 172.21.15.93:0/162778818 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f046c01b030 con 0x7f047c0f55a0 2021-11-20T08:57:04.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.223+0000 7f0478ff9700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f046405c2a0 0x7f046405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.223+0000 7f0478ff9700 1 -- 172.21.15.93:0/162778818 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f046c075c70 con 0x7f047c0f55a0 2021-11-20T08:57:04.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.223+0000 7f047b7fe700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f046405c2a0 0x7f046405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.224+0000 7f047b7fe700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f046405c2a0 0x7f046405e760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0470005010 tx=0x7f0470016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:04.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.228+0000 7f0478ff9700 1 -- 172.21.15.93:0/162778818 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f046c0489e0 con 0x7f047c0f55a0 2021-11-20T08:57:04.374 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.375+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f047c10f780 con 0x7f047c0f55a0 2021-11-20T08:57:04.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.376+0000 7f0478ff9700 1 -- 172.21.15.93:0/162778818 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+14086 (secure 0 0 0) 0x7f046c019020 con 0x7f047c0f55a0 2021-11-20T08:57:04.375 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:57:04.376 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":43,"fsid":"11458b34-49df-11ec-8c2c-001a4aab830c","created":"2021-11-20T08:51:57.464444+0000","modified":"2021-11-20T08:56:44.326907+0000","last_up_change":"2021-11-20T08:55:49.151657+0000","last_in_change":"2021-11-20T08:55:32.747237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T08:55:09.134409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T08:56:41.253240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T08:56:41.551098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b62eaab5-818e-4739-98c3-d386425d4837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6803","nonce":1548414389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6805","nonce":1548414389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6809","nonce":1548414389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1548414389},{"type":"v1","addr":"172.21.15.93:6807","nonce":1548414389}]},"public_addr":"172.21.15.93:6803/1548414389","cluster_addr":"172.21.15.93:6805/1548414389","heartbeat_back_addr":"172.21.15.93:6809/1548414389","heartbeat_front_addr":"172.21.15.93:6807/1548414389","state":["exists","up"]},{"osd":1,"uuid":"a46f3336-3bc4-4736-9095-c4d72e662323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6811","nonce":3785015356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6813","nonce":3785015356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6817","nonce":3785015356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":3785015356},{"type":"v1","addr":"172.21.15.93:6815","nonce":3785015356}]},"public_addr":"172.21.15.93:6811/3785015356","cluster_addr":"172.21.15.93:6813/3785015356","heartbeat_back_addr":"172.21.15.93:6817/3785015356","heartbeat_front_addr":"172.21.15.93:6815/3785015356","state":["exists","up"]},{"osd":2,"uuid":"72096543-af61-429d-a8d1-a77726a1f3eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6819","nonce":82486730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6821","nonce":82486730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6825","nonce":82486730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":82486730},{"type":"v1","addr":"172.21.15.93:6823","nonce":82486730}]},"public_addr":"172.21.15.93:6819/82486730","cluster_addr":"172.21.15.93:6821/82486730","heartbeat_back_addr":"172.21.15.93:6825/82486730","heartbeat_front_addr":"172.21.15.93:6823/82486730","state":["exists","up"]},{"osd":3,"uuid":"37ba4275-7023-4f74-ba1f-7ad2f864476f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6801","nonce":1093924688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6803","nonce":1093924688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6807","nonce":1093924688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1093924688},{"type":"v1","addr":"172.21.15.102:6805","nonce":1093924688}]},"public_addr":"172.21.15.102:6801/1093924688","cluster_addr":"172.21.15.102:6803/1093924688","heartbeat_back_addr":"172.21.15.102:6807/1093924688","heartbeat_front_addr":"172.21.15.102:6805/1093924688","state":["exists","up"]},{"osd":4,"uuid":"c8db5c60-20bb-4b21-b5c8-ae07c9f4fcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6809","nonce":2700333748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6811","nonce":2700333748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6815","nonce":2700333748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":2700333748},{"type":"v1","addr":"172.21.15.102:6813","nonce":2700333748}]},"public_addr":"172.21.15.102:6809/2700333748","cluster_addr":"172.21.15.102:6811/2700333748","heartbeat_back_addr":"172.21.15.102:6815/2700333748","heartbeat_front_addr":"172.21.15.102:6813/2700333748","state":["exists","up"]},{"osd":5,"uuid":"fab60808-4ace-4f26-9c42-f72340134a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6817","nonce":4083093586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6819","nonce":4083093586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6823","nonce":4083093586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4083093586},{"type":"v1","addr":"172.21.15.102:6821","nonce":4083093586}]},"public_addr":"172.21.15.102:6817/4083093586","cluster_addr":"172.21.15.102:6819/4083093586","heartbeat_back_addr":"172.21.15.102:6823/4083093586","heartbeat_front_addr":"172.21.15.102:6821/4083093586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:21.010676+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:41.314338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:54:57.961372+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:13.733626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:30.250474+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T08:55:47.331557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:0/3689853676":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/3631043468":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:0/485709266":"2021-11-21T08:52:37.986811+0000","172.21.15.93:6800/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1880974768":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/1518372611":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/3294416137":"2021-11-21T08:52:51.424660+0000","172.21.15.93:6801/3748247363":"2021-11-21T08:52:37.986811+0000","172.21.15.93:0/2364704567":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6800/1580490298":"2021-11-21T08:52:15.847551+0000","172.21.15.93:6801/1580490298":"2021-11-21T08:52:15.847551+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T08:57:04.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f046405c2a0 msgr2=0x7f046405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f046405c2a0 0x7f046405e760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0470005010 tx=0x7f0470016040).stop 2021-11-20T08:57:04.378 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f55a0 msgr2=0x7f047c0f9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.378 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f55a0 0x7f047c0f9520 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f046c00e7c0 tx=0x7f046c00b040).stop 2021-11-20T08:57:04.378 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 shutdown_connections 2021-11-20T08:57:04.378 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f046405c2a0 0x7f046405e760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f047c0f55a0 0x7f047c0f9520 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 --2- 172.21.15.93:0/162778818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f047c0f6840 0x7f047c0f7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 >> 172.21.15.93:0/162778818 conn(0x7f047c0f0ce0 msgr2=0x7f047c05fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:04.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 shutdown_connections 2021-11-20T08:57:04.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.378+0000 7f04822b7700 1 -- 172.21.15.93:0/162778818 wait complete. 2021-11-20T08:57:04.389 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T08:55:09.134409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T08:56:41.253240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T08:56:41.551098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T08:57:04.390 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T08:57:04.555 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/1871928691 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 msgr2=0x7ff3d40f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.556 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/1871928691 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d40f6af0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ff3c8004660 tx=0x7ff3c8009b30).stop 2021-11-20T08:57:04.556 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/1871928691 shutdown_connections 2021-11-20T08:57:04.556 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/1871928691 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d40f6af0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/1871928691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3d40f53f0 0x7ff3d40f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/1871928691 >> 172.21.15.93:0/1871928691 conn(0x7ff3d40f0b30 msgr2=0x7ff3d40f2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:04.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/1871928691 shutdown_connections 2021-11-20T08:57:04.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/1871928691 wait complete. 2021-11-20T08:57:04.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.553+0000 7ff3d8c6f700 1 Processor -- start 2021-11-20T08:57:04.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d8c6f700 1 -- start start 2021-11-20T08:57:04.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d8c6f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3d40f53f0 0x7ff3d4109a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d8c6f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d410bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d8c6f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40603b0 con 0x7ff3d40f6690 2021-11-20T08:57:04.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d8c6f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40604f0 con 0x7ff3d40f53f0 2021-11-20T08:57:04.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d2ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d410bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d2ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d410bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43242/0 (socket says 172.21.15.93:43242) 2021-11-20T08:57:04.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d2ffd700 1 -- 172.21.15.93:0/2385225745 learned_addr learned my addr 172.21.15.93:0/2385225745 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:04.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d2ffd700 1 -- 172.21.15.93:0/2385225745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3d40f53f0 msgr2=0x7ff3d4109a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d2ffd700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3d40f53f0 0x7ff3d4109a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:57:04.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d2ffd700 1 -- 172.21.15.93:0/2385225745 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c8005040 con 0x7ff3d40f6690 2021-11-20T08:57:04.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3d2ffd700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d410bf90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff3c8009830 tx=0x7ff3c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:04.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.554+0000 7ff3bbfff700 1 -- 172.21.15.93:0/2385225745 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c8018070 con 0x7ff3d40f6690 2021-11-20T08:57:04.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.555+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d410c4e0 con 0x7ff3d40f6690 2021-11-20T08:57:04.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.555+0000 7ff3bbfff700 1 -- 172.21.15.93:0/2385225745 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3c8013410 con 0x7ff3d40f6690 2021-11-20T08:57:04.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.555+0000 7ff3bbfff700 1 -- 172.21.15.93:0/2385225745 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c801c620 con 0x7ff3d40f6690 2021-11-20T08:57:04.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.555+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d410ca40 con 0x7ff3d40f6690 2021-11-20T08:57:04.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.555+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d4045ad0 con 0x7ff3d40f6690 2021-11-20T08:57:04.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.556+0000 7ff3bbfff700 1 -- 172.21.15.93:0/2385225745 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff3c801c8e0 con 0x7ff3d40f6690 2021-11-20T08:57:04.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.556+0000 7ff3bbfff700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3bc05c2b0 0x7ff3bc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.556+0000 7ff3bbfff700 1 -- 172.21.15.93:0/2385225745 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7ff3c8077da0 con 0x7ff3d40f6690 2021-11-20T08:57:04.567 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.556+0000 7ff3d37fe700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3bc05c2b0 0x7ff3bc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.567 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.557+0000 7ff3d37fe700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3bc05c2b0 0x7ff3bc05e770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ff3d4060030 tx=0x7ff3c4004960).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:04.568 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.560+0000 7ff3bbfff700 1 -- 172.21.15.93:0/2385225745 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3c804aa00 con 0x7ff3d40f6690 2021-11-20T08:57:04.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.711+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff3d40fdcc0 con 0x7ff3d40f6690 2021-11-20T08:57:04.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.711+0000 7ff3bbfff700 1 -- 172.21.15.93:0/2385225745 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v43) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff3c804e020 con 0x7ff3d40f6690 2021-11-20T08:57:04.712 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 1 2021-11-20T08:57:04.714 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3bc05c2b0 msgr2=0x7ff3bc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.715 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3bc05c2b0 0x7ff3bc05e770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ff3d4060030 tx=0x7ff3c4004960).stop 2021-11-20T08:57:04.715 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 msgr2=0x7ff3d410bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.715 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d410bf90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff3c8009830 tx=0x7ff3c8009b30).stop 2021-11-20T08:57:04.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 shutdown_connections 2021-11-20T08:57:04.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3bc05c2b0 0x7ff3bc05e770 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3d40f53f0 0x7ff3d4109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:57:04.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 --2- 172.21.15.93:0/2385225745 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3d40f6690 0x7ff3d410bf90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 >> 172.21.15.93:0/2385225745 conn(0x7ff3d40f0b30 msgr2=0x7ff3d40fa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:04.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 shutdown_connections 2021-11-20T08:57:04.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.713+0000 7ff3d8c6f700 1 -- 172.21.15.93:0/2385225745 wait complete. 2021-11-20T08:57:04.724 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T08:57:04.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- 172.21.15.93:0/3140315242 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 msgr2=0x7fb9380f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 --2- 172.21.15.93:0/3140315242 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb9380f59a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fb9280099e0 tx=0x7fb928004d10).stop 2021-11-20T08:57:04.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- 172.21.15.93:0/3140315242 shutdown_connections 2021-11-20T08:57:04.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 --2- 172.21.15.93:0/3140315242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9380f6820 0x7fb9380f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 --2- 172.21.15.93:0/3140315242 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb9380f59a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- 172.21.15.93:0/3140315242 >> 172.21.15.93:0/3140315242 conn(0x7fb9380f0ce0 msgr2=0x7fb9380f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:04.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- 172.21.15.93:0/3140315242 shutdown_connections 2021-11-20T08:57:04.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- 172.21.15.93:0/3140315242 wait complete. 2021-11-20T08:57:04.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 Processor -- start 2021-11-20T08:57:04.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- start start 2021-11-20T08:57:04.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb938105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9380f6820 0x7fb9381063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938106950 con 0x7fb9380f5580 2021-11-20T08:57:04.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.843+0000 7fb93fa84700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938106a90 con 0x7fb9380f6820 2021-11-20T08:57:04.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d820700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb938105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d820700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb938105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43250/0 (socket says 172.21.15.93:43250) 2021-11-20T08:57:04.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d820700 1 -- 172.21.15.93:0/4157842917 learned_addr learned my addr 172.21.15.93:0/4157842917 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:04.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d01f700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9380f6820 0x7fb9381063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d820700 1 -- 172.21.15.93:0/4157842917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9380f6820 msgr2=0x7fb9381063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:04.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d820700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9380f6820 0x7fb9381063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:04.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d820700 1 -- 172.21.15.93:0/4157842917 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928009640 con 0x7fb9380f5580 2021-11-20T08:57:04.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93d820700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb938105eb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fb928007830 tx=0x7fb928004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:04.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb9367fc700 1 -- 172.21.15.93:0/4157842917 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92800e030 con 0x7fb9380f5580 2021-11-20T08:57:04.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93810b4d0 con 0x7fb9380f5580 2021-11-20T08:57:04.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb9367fc700 1 -- 172.21.15.93:0/4157842917 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb92801d8d0 con 0x7fb9380f5580 2021-11-20T08:57:04.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb9367fc700 1 -- 172.21.15.93:0/4157842917 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb928018070 con 0x7fb9380f5580 2021-11-20T08:57:04.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.844+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93810ba10 con 0x7fb9380f5580 2021-11-20T08:57:04.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.845+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb938027a00 con 0x7fb9380f5580 2021-11-20T08:57:04.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.845+0000 7fb9367fc700 1 -- 172.21.15.93:0/4157842917 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb92801da40 con 0x7fb9380f5580 2021-11-20T08:57:04.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.845+0000 7fb9367fc700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb92005c2a0 0x7fb92005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:04.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.846+0000 7fb9367fc700 1 -- 172.21.15.93:0/4157842917 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fb928077e30 con 0x7fb9380f5580 2021-11-20T08:57:04.852 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.846+0000 7fb93d01f700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb92005c2a0 0x7fb92005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:04.852 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.846+0000 7fb93d01f700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb92005c2a0 0x7fb92005e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb92c0041d0 tx=0x7fb92c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:04.852 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.849+0000 7fb9367fc700 1 -- 172.21.15.93:0/4157842917 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb92804aaf0 con 0x7fb9380f5580 2021-11-20T08:57:04.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.997+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fb9380fdd10 con 0x7fb9380f5580 2021-11-20T08:57:04.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.997+0000 7fb9367fc700 1 -- 172.21.15.93:0/4157842917 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb92804e110 con 0x7fb9380f5580 2021-11-20T08:57:04.998 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 32 2021-11-20T08:57:05.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb92005c2a0 msgr2=0x7fb92005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb92005c2a0 0x7fb92005e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb92c0041d0 tx=0x7fb92c009040).stop 2021-11-20T08:57:05.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 msgr2=0x7fb938105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb938105eb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fb928007830 tx=0x7fb928004d10).stop 2021-11-20T08:57:05.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 shutdown_connections 2021-11-20T08:57:05.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb92005c2a0 0x7fb92005e760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb9380f5580 0x7fb938105eb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 --2- 172.21.15.93:0/4157842917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9380f6820 0x7fb9381063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 >> 172.21.15.93:0/4157842917 conn(0x7fb9380f0ce0 msgr2=0x7fb9380fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:05.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 shutdown_connections 2021-11-20T08:57:05.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:04.999+0000 7fb93fa84700 1 -- 172.21.15.93:0/4157842917 wait complete. 2021-11-20T08:57:05.009 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:04 smithi093 conmon[33522]: audit 2021-11-20T08:57:04.376673+0000 mon.smithi093 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.93:0/162778818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:57:05.009 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:04 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:57:05.010 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:04 smithi093 conmon[33522]: 08:57:04.712422+0000 mon.smithi093 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.93:0/2385225745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:57:05.011 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T08:57:05.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1803060168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f4d90 msgr2=0x7fcef80f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1803060168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f4d90 0x7fcef80f51b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fcee8004660 tx=0x7fcee8009b30).stop 2021-11-20T08:57:05.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1803060168 shutdown_connections 2021-11-20T08:57:05.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1803060168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcef80f5f90 0x7fcef80f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1803060168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f4d90 0x7fcef80f51b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1803060168 >> 172.21.15.93:0/1803060168 conn(0x7fcef80f04d0 msgr2=0x7fcef80f2910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:05.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1803060168 shutdown_connections 2021-11-20T08:57:05.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1803060168 wait complete. 2021-11-20T08:57:05.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 Processor -- start 2021-11-20T08:57:05.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.134+0000 7fcefcc8c700 1 -- start start 2021-11-20T08:57:05.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcefcc8c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcef80f4d90 0x7fcef8106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcefcc8c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f5f90 0x7fcef81073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcefcc8c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8107970 con 0x7fcef80f5f90 2021-11-20T08:57:05.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcefcc8c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8107ab0 con 0x7fcef80f4d90 2021-11-20T08:57:05.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef5d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f5f90 0x7fcef81073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef5d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f5f90 0x7fcef81073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43264/0 (socket says 172.21.15.93:43264) 2021-11-20T08:57:05.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef5d9b700 1 -- 172.21.15.93:0/1337060944 learned_addr learned my addr 172.21.15.93:0/1337060944 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:05.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef659c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcef80f4d90 0x7fcef8106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef5d9b700 1 -- 172.21.15.93:0/1337060944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcef80f4d90 msgr2=0x7fcef8106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef5d9b700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcef80f4d90 0x7fcef8106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef5d9b700 1 -- 172.21.15.93:0/1337060944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8005040 con 0x7fcef80f5f90 2021-11-20T08:57:05.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcef5d9b700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f5f90 0x7fcef81073c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fceec004870 tx=0x7fceec00c040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcee6ffd700 1 -- 172.21.15.93:0/1337060944 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceec00f040 con 0x7fcef80f5f90 2021-11-20T08:57:05.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcee6ffd700 1 -- 172.21.15.93:0/1337060944 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fceec00fb40 con 0x7fcef80f5f90 2021-11-20T08:57:05.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef810c550 con 0x7fcef80f5f90 2021-11-20T08:57:05.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcee6ffd700 1 -- 172.21.15.93:0/1337060944 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceec015ca0 con 0x7fcef80f5f90 2021-11-20T08:57:05.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.135+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef810cb10 con 0x7fcef80f5f90 2021-11-20T08:57:05.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.136+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef8027a00 con 0x7fcef80f5f90 2021-11-20T08:57:05.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.137+0000 7fcee6ffd700 1 -- 172.21.15.93:0/1337060944 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fceec01c060 con 0x7fcef80f5f90 2021-11-20T08:57:05.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.137+0000 7fcee6ffd700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcee005c2a0 0x7fcee005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.137+0000 7fcee6ffd700 1 -- 172.21.15.93:0/1337060944 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fceec075ff0 con 0x7fcef80f5f90 2021-11-20T08:57:05.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.137+0000 7fcef659c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcee005c2a0 0x7fcee005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.137+0000 7fcef659c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcee005c2a0 0x7fcee005e760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcee8000c00 tx=0x7fcee8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.141+0000 7fcee6ffd700 1 -- 172.21.15.93:0/1337060944 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fceec048cb0 con 0x7fcef80f5f90 2021-11-20T08:57:05.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:04 smithi102 conmon[36472]: audit 2021-11-20T08:57:04.376673+0000 mon.smithi093 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.93:0/162778818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T08:57:05.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:04 smithi102 conmon[36472]: audit 2021-11-20T08:57:04.712422+0000 mon.smithi093 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.93:0/2385225745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T08:57:05.288 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.289+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fcef80fcc30 con 0x7fcef80f5f90 2021-11-20T08:57:05.289 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.289+0000 7fcee6ffd700 1 -- 172.21.15.93:0/1337060944 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7fceec04c2d0 con 0x7fcef80f5f90 2021-11-20T08:57:05.290 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 32 2021-11-20T08:57:05.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcee005c2a0 msgr2=0x7fcee005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcee005c2a0 0x7fcee005e760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcee8000c00 tx=0x7fcee8009b30).stop 2021-11-20T08:57:05.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f5f90 msgr2=0x7fcef81073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f5f90 0x7fcef81073c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fceec004870 tx=0x7fceec00c040).stop 2021-11-20T08:57:05.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 shutdown_connections 2021-11-20T08:57:05.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcee005c2a0 0x7fcee005e760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcef80f4d90 0x7fcef8106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 --2- 172.21.15.93:0/1337060944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcef80f5f90 0x7fcef81073c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 >> 172.21.15.93:0/1337060944 conn(0x7fcef80f04d0 msgr2=0x7fcef80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:05.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 shutdown_connections 2021-11-20T08:57:05.294 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.291+0000 7fcefcc8c700 1 -- 172.21.15.93:0/1337060944 wait complete. 2021-11-20T08:57:05.302 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T08:57:05.303 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:57:05.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 -- 172.21.15.93:0/2528662782 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 msgr2=0x7fbc5c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/2528662782 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c0f6bd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbc4c004660 tx=0x7fbc4c009b30).stop 2021-11-20T08:57:05.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 -- 172.21.15.93:0/2528662782 shutdown_connections 2021-11-20T08:57:05.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/2528662782 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c0f6bd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/2528662782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc5c0f54d0 0x7fbc5c0f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:57:05.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 -- 172.21.15.93:0/2528662782 >> 172.21.15.93:0/2528662782 conn(0x7fbc5c0f0c10 msgr2=0x7fbc5c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:05.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 -- 172.21.15.93:0/2528662782 shutdown_connections 2021-11-20T08:57:05.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.417+0000 7fbc60bf6700 1 -- 172.21.15.93:0/2528662782 wait complete. 2021-11-20T08:57:05.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc60bf6700 1 Processor -- start 2021-11-20T08:57:05.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc60bf6700 1 -- start start 2021-11-20T08:57:05.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc60bf6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc5c0f54d0 0x7fbc5c105cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc60bf6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c1061c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc60bf6700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c106800 con 0x7fbc5c0f6770 2021-11-20T08:57:05.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc60bf6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c106f70 con 0x7fbc5c0f54d0 2021-11-20T08:57:05.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc5affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c1061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc5affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c1061c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43274/0 (socket says 172.21.15.93:43274) 2021-11-20T08:57:05.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.418+0000 7fbc5affd700 1 -- 172.21.15.93:0/3874147528 learned_addr learned my addr 172.21.15.93:0/3874147528 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:05.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc5b7fe700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc5c0f54d0 0x7fbc5c105cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc5affd700 1 -- 172.21.15.93:0/3874147528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc5c0f54d0 msgr2=0x7fbc5c105cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc5affd700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc5c0f54d0 0x7fbc5c105cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc5affd700 1 -- 172.21.15.93:0/3874147528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c005040 con 0x7fbc5c0f6770 2021-11-20T08:57:05.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc5affd700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c1061c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fbc4c009830 tx=0x7fbc4c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc53fff700 1 -- 172.21.15.93:0/3874147528 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc4c018070 con 0x7fbc5c0f6770 2021-11-20T08:57:05.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc53fff700 1 -- 172.21.15.93:0/3874147528 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc4c013410 con 0x7fbc5c0f6770 2021-11-20T08:57:05.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c1071f0 con 0x7fbc5c0f6770 2021-11-20T08:57:05.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc53fff700 1 -- 172.21.15.93:0/3874147528 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc4c01c620 con 0x7fbc5c0f6770 2021-11-20T08:57:05.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.419+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c107750 con 0x7fbc5c0f6770 2021-11-20T08:57:05.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.420+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c045ad0 con 0x7fbc5c0f6770 2021-11-20T08:57:05.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.420+0000 7fbc53fff700 1 -- 172.21.15.93:0/3874147528 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbc4c01c890 con 0x7fbc5c0f6770 2021-11-20T08:57:05.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.421+0000 7fbc53fff700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbc4805c300 0x7fbc4805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.430 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.421+0000 7fbc53fff700 1 -- 172.21.15.93:0/3874147528 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fbc4c077cb0 con 0x7fbc5c0f6770 2021-11-20T08:57:05.430 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.421+0000 7fbc5b7fe700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbc4805c300 0x7fbc4805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.430 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.421+0000 7fbc5b7fe700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbc4805c300 0x7fbc4805e7c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fbc5c0f65d0 tx=0x7fbc44008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.431 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.425+0000 7fbc53fff700 1 -- 172.21.15.93:0/3874147528 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc4c04a990 con 0x7fbc5c0f6770 2021-11-20T08:57:05.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.609+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc5c027a00 con 0x7fbc5c0f6770 2021-11-20T08:57:05.609 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.610+0000 7fbc53fff700 1 -- 172.21.15.93:0/3874147528 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2932 (secure 0 0 0) 0x7fbc5c027a00 con 0x7fbc5c0f6770 2021-11-20T08:57:05.609 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:57:05.610 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14428,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6827/2783208877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":2783208877},{"type":"v1","addr":"172.21.15.93:6827","nonce":2783208877}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":10},{"gid":14438,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/1976124814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":1976124814},{"type":"v1","addr":"172.21.15.93:6829","nonce":1976124814}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":12},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:56:50.485290+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24257":{"gid":24257,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.102:6825/1423063558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1423063558},{"type":"v1","addr":"172.21.15.102:6825","nonce":1423063558}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:57:05.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.611+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbc4805c300 msgr2=0x7fbc4805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.611+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbc4805c300 0x7fbc4805e7c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fbc5c0f65d0 tx=0x7fbc44008040).stop 2021-11-20T08:57:05.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.611+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 msgr2=0x7fbc5c1061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.611+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c1061c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fbc4c009830 tx=0x7fbc4c009b30).stop 2021-11-20T08:57:05.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.612+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 shutdown_connections 2021-11-20T08:57:05.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.612+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbc4805c300 0x7fbc4805e7c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.612+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc5c0f54d0 0x7fbc5c105cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.612+0000 7fbc60bf6700 1 --2- 172.21.15.93:0/3874147528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc5c0f6770 0x7fbc5c1061c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.612+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 >> 172.21.15.93:0/3874147528 conn(0x7fbc5c0f0c10 msgr2=0x7fbc5c0fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:05.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.612+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 shutdown_connections 2021-11-20T08:57:05.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.612+0000 7fbc60bf6700 1 -- 172.21.15.93:0/3874147528 wait complete. 2021-11-20T08:57:05.615 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 12 2021-11-20T08:57:05.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:56:50.485290+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24257': {'gid': 24257, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.102:6825/1423063558', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1423063558}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1423063558}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:57:05.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-11-20T08:57:05.624 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi102.lhrlmh status 2021-11-20T08:57:05.755 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 -- 172.21.15.93:0/2668841995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82a40602a0 msgr2=0x7f82a40fb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 --2- 172.21.15.93:0/2668841995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82a40602a0 0x7f82a40fb510 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f8298004660 tx=0x7f8298009b30).stop 2021-11-20T08:57:05.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 -- 172.21.15.93:0/2668841995 shutdown_connections 2021-11-20T08:57:05.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 --2- 172.21.15.93:0/2668841995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82a40fba00 0x7f82a40fddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 --2- 172.21.15.93:0/2668841995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82a40602a0 0x7f82a40fb510 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 -- 172.21.15.93:0/2668841995 >> 172.21.15.93:0/2668841995 conn(0x7f82a40f0b10 msgr2=0x7f82a40f2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:05.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 -- 172.21.15.93:0/2668841995 shutdown_connections 2021-11-20T08:57:05.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.740+0000 7f82a89ce700 1 -- 172.21.15.93:0/2668841995 wait complete. 2021-11-20T08:57:05.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a89ce700 1 Processor -- start 2021-11-20T08:57:05.759 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a89ce700 1 -- start start 2021-11-20T08:57:05.759 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a89ce700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82a40602a0 0x7f82a4108f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.760 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a89ce700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82a40fba00 0x7f82a4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.760 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a89ce700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a4109a00 con 0x7f82a40fba00 2021-11-20T08:57:05.760 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a89ce700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a4109b40 con 0x7f82a40602a0 2021-11-20T08:57:05.761 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a2ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82a40fba00 0x7f82a4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.761 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a2ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82a40fba00 0x7f82a4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43284/0 (socket says 172.21.15.93:43284) 2021-11-20T08:57:05.762 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a2ffd700 1 -- 172.21.15.93:0/4216706630 learned_addr learned my addr 172.21.15.93:0/4216706630 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:05.762 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.741+0000 7f82a37fe700 1 --2- 172.21.15.93:0/4216706630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82a40602a0 0x7f82a4108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.762 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f82a2ffd700 1 -- 172.21.15.93:0/4216706630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82a40602a0 msgr2=0x7f82a4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.763 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f82a2ffd700 1 --2- 172.21.15.93:0/4216706630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82a40602a0 0x7f82a4108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.763 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f82a2ffd700 1 -- 172.21.15.93:0/4216706630 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8298005040 con 0x7f82a40fba00 2021-11-20T08:57:05.763 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f82a2ffd700 1 --2- 172.21.15.93:0/4216706630 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82a40fba00 0x7f82a4109450 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f829400c510 tx=0x7f8294007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.763 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f828bfff700 1 -- 172.21.15.93:0/4216706630 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f829400f650 con 0x7f82a40fba00 2021-11-20T08:57:05.764 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f828bfff700 1 -- 172.21.15.93:0/4216706630 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f829400fc60 con 0x7f82a40fba00 2021-11-20T08:57:05.764 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f82a89ce700 1 -- 172.21.15.93:0/4216706630 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a410a4e0 con 0x7f82a40fba00 2021-11-20T08:57:05.764 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f828bfff700 1 -- 172.21.15.93:0/4216706630 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8294012c10 con 0x7f82a40fba00 2021-11-20T08:57:05.764 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.742+0000 7f82a89ce700 1 -- 172.21.15.93:0/4216706630 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a410aaa0 con 0x7f82a40fba00 2021-11-20T08:57:05.765 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.743+0000 7f828bfff700 1 -- 172.21.15.93:0/4216706630 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8294010070 con 0x7f82a40fba00 2021-11-20T08:57:05.765 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.743+0000 7f828bfff700 1 --2- 172.21.15.93:0/4216706630 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f828c05c2b0 0x7f828c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.766 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.743+0000 7f828bfff700 1 -- 172.21.15.93:0/4216706630 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8294075b70 con 0x7f82a40fba00 2021-11-20T08:57:05.766 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.744+0000 7f82a37fe700 1 --2- 172.21.15.93:0/4216706630 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f828c05c2b0 0x7f828c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.766 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.744+0000 7f82a37fe700 1 --2- 172.21.15.93:0/4216706630 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f828c05c2b0 0x7f828c05e770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8298004630 tx=0x7f8298009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.767 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.745+0000 7f82bb7dbb80 1 Processor -- start 2021-11-20T08:57:05.767 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.745+0000 7f82bb7dbb80 1 -- start start 2021-11-20T08:57:05.767 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.745+0000 7f82bb7dbb80 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb78a0 0x561580cb7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.767 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.745+0000 7f82bb7dbb80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb81b0 0x561580cb8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82bb7dbb80 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x561580bf64d0 con 0x561580cb78a0 2021-11-20T08:57:05.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82bb7dbb80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x561580c0a3e0 con 0x561580cb81b0 2021-11-20T08:57:05.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb78a0 0x561580cb7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a37fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb78a0 0x561580cb7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43288/0 (socket says 172.21.15.93:43288) 2021-11-20T08:57:05.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a37fe700 1 -- 172.21.15.93:0/420153191 learned_addr learned my addr 172.21.15.93:0/420153191 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:05.769 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a37fe700 1 -- 172.21.15.93:0/420153191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb81b0 msgr2=0x561580cb8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T08:57:05.769 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a2ffd700 1 --2- 172.21.15.93:0/420153191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb81b0 0x561580cb8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.769 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a37fe700 1 --2- 172.21.15.93:0/420153191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb81b0 0x561580cb8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.769 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a37fe700 1 -- 172.21.15.93:0/420153191 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561580c863c0 con 0x561580cb78a0 2021-11-20T08:57:05.770 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a2ffd700 1 --2- 172.21.15.93:0/420153191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb81b0 0x561580cb8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T08:57:05.770 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82a37fe700 1 --2- 172.21.15.93:0/420153191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb78a0 0x561580cb7cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f8298004600 tx=0x7f829801d000).ready entity=mon.0 client_cookie=e49e24e3829ad0e7 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.770 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82897fa700 1 -- 172.21.15.93:0/420153191 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f829800e040 con 0x561580cb78a0 2021-11-20T08:57:05.770 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82897fa700 1 -- 172.21.15.93:0/420153191 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8298019040 con 0x561580cb78a0 2021-11-20T08:57:05.771 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.746+0000 7f82897fa700 1 -- 172.21.15.93:0/420153191 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8298029c20 con 0x561580cb78a0 2021-11-20T08:57:05.771 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/420153191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb78a0 msgr2=0x561580cb7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:57:05.771 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 --2- 172.21.15.93:0/420153191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb78a0 0x561580cb7cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f8298004600 tx=0x7f829801d000).stop 2021-11-20T08:57:05.772 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/420153191 shutdown_connections 2021-11-20T08:57:05.772 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 --2- 172.21.15.93:0/420153191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb81b0 0x561580cb8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.772 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 --2- 172.21.15.93:0/420153191 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb78a0 0x561580cb7cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:57:05.773 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/420153191 >> 172.21.15.93:0/420153191 conn(0x561580b251d0 msgr2=0x561580cb2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:57:05.773 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/420153191 shutdown_connections 2021-11-20T08:57:05.773 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/420153191 wait complete. 2021-11-20T08:57:05.774 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 -- start start 2021-11-20T08:57:05.774 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 Processor -- start 2021-11-20T08:57:05.774 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb78a0 0x561580cc0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.774 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb81b0 0x561580cc0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.747+0000 7f82bb7dbb80 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x561580c0a520 con 0x561580cb81b0 2021-11-20T08:57:05.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x561580c0aaa0 con 0x561580cb78a0 2021-11-20T08:57:05.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 10 client.0 ll_register_callbacks cb 0x561580c778b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T08:57:05.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82a2ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb81b0 0x561580cc0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:05.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82a2ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb81b0 0x561580cc0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43294/0 (socket says 172.21.15.93:43294) 2021-11-20T08:57:05.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82a2ffd700 1 -- 172.21.15.93:0/3652382463 learned_addr learned my addr 172.21.15.93:0/3652382463 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:57:05.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82a2ffd700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb78a0 msgr2=0x561580cc0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T08:57:05.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82a2ffd700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561580cb78a0 0x561580cc0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:57:05.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82a2ffd700 1 -- 172.21.15.93:0/3652382463 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a40fe590 con 0x561580cb81b0 2021-11-20T08:57:05.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82a2ffd700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561580cb81b0 0x561580cc0fb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f8294004370 tx=0x7f8294007f10).ready entity=mon.0 client_cookie=c7c4aa536205540a server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:57:05.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82827fc700 10 client.0 ms_handle_connect on v2:172.21.15.93:3300/0 2021-11-20T08:57:05.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82827fc700 1 -- 172.21.15.93:0/3652382463 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82940484c0 con 0x561580cb81b0 2021-11-20T08:57:05.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82827fc700 1 -- 172.21.15.93:0/3652382463 <== mon.0 v2:172.21.15.93:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8294048620 con 0x561580cb81b0 2021-11-20T08:57:05.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/3652382463 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561580c85ce0 con 0x561580cb81b0 2021-11-20T08:57:05.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82827fc700 1 -- 172.21.15.93:0/3652382463 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8294050ac0 con 0x561580cb81b0 2021-11-20T08:57:05.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 10 client.14532 fetch_fsmap learned FSMap version 12 2021-11-20T08:57:05.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/3652382463 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=3+}) v3 -- 0x561580bf9630 con 0x561580cb81b0 2021-11-20T08:57:05.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82827fc700 1 -- 172.21.15.93:0/3652382463 <== mon.0 v2:172.21.15.93:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x561580c85ce0 con 0x561580cb81b0 2021-11-20T08:57:05.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82827fc700 1 -- 172.21.15.93:0/3652382463 <== mon.0 v2:172.21.15.93:3300/0 5 ==== fsmap(e 12) v1 ==== 1724+0+0 (secure 0 0 0) 0x7f829404b450 con 0x561580cb81b0 2021-11-20T08:57:05.781 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 10 client.14532 fetch_fsmap finished waiting for FSMap version 12 2021-11-20T08:57:05.781 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 10 client.14532 resolve_mds: resolved name 'cephfs.smithi102.lhrlmh' to daemon mds.cephfs.smithi102.lhrlmh 2021-11-20T08:57:05.781 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 20 client.14532 populate_metadata read hostname 'smithi093' 2021-11-20T08:57:05.782 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.748+0000 7f82bb7dbb80 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T08:57:05.782 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.749+0000 7f82bb7dbb80 4 client.14532 mds_command: new command op to 24257 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T08:57:05.783 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.749+0000 7f82bb7dbb80 1 -- 172.21.15.93:0/3652382463 --> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561580c00df0 con 0x561580cc97a0 2021-11-20T08:57:05.783 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.749+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:05.783 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.749+0000 7f82827fc700 1 client.14532 ms_handle_refused on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:05.784 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.749+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T08:57:05.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.950+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:05.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.950+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T08:57:05.949 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:05.950+0000 7f82827fc700 1 client.14532 ms_handle_refused on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: audit 2021-11-20T08:57:04.998417+0000 mon.smithi093 (mon 2021-11-20T08:57:05.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: .0) 536 : audit [DBG] from='client.? 172.21.15.93:0/4157842917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T08:57:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: audit 2021-11-20T08:57:05.290603+0000 mon.smithi093 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.93:0/1337060944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T08:57:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: cluster 2021-11-20T08:57:05.526211 2021-11-20T08:57:05.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 196 : cluster [DBG] 2021-11-20T08:57:05.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: pgmap v159: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:57:05.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:57:05.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: 57:05.610938+0000 mon.smithi093 (mon.0 2021-11-20T08:57:05.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:05 smithi093 conmon[33522]: ) 538 : audit [DBG] from='client.? 172.21.15.93:0/3874147528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:57:06.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:05 smithi102 conmon[36472]: audit 2021-11-20T08:57:04.998417+0000 mon.smithi093 (mon.0) 536 : audit [DBG] 2021-11-20T08:57:06.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:05 smithi102 conmon[36472]: from='client.? 172.21.15.93:0/4157842917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T08:57:06.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:05 smithi102 conmon[36472]: audit 2021-11-20T08:57:05.290603+0000 mon.smithi093 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.93:0/1337060944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T08:57:06.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:05 smithi102 conmon[36472]: cluster 2021-11-20T08:57:05.526211+0000 mgr.smithi093.qdikkm (mgr.14162) 196 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:57:06.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:05 smithi102 conmon[36472]: audit 2021-11-20T08:57:05.610938+0000 mon.smithi093 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.93:0/3874147528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:57:06.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:06.351+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:06.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:06.351+0000 7f82827fc700 1 client.14532 ms_handle_refused on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:06.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:06.351+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T08:57:07.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:07.151+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:07.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:07.151+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T08:57:07.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:07.151+0000 7f82827fc700 1 client.14532 ms_handle_refused on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:07.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:07 smithi102 conmon[36472]: cluster 2021-11-20T08:57:07.526937+0000 mgr.smithi093.qdikkm (mgr.14162) 197 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:07.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:07 smithi093 conmon[33522]: cluster 2021-11-20T08:57:07.526937+0000 mgr.smithi093.qdikkm (mgr.14162) 197 : 2021-11-20T08:57:07.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:07 smithi093 conmon[33522]: cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:08.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:08.753+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:08.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:08.753+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T08:57:08.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:08.753+0000 7f82827fc700 1 client.14532 ms_handle_refused on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:10.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:10 smithi102 conmon[36472]: cluster 2021-11-20T08:57:09.527509+0000 mgr.smithi093.qdikkm (mgr.14162) 198 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:10.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:10 smithi102 conmon[36472]: 2021-11-20T08:57:10.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:10 smithi093 conmon[33522]: cluster 2021-11-20T08:57:09.527509+0000 mgr.smithi093.qdikkm (mgr.14162) 198 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:11.956 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:11.957+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:11.956 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:11.957+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 6.400000 2021-11-20T08:57:11.957 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:11.957+0000 7f82827fc700 1 client.14532 ms_handle_refused on v2:172.21.15.102:6824/1423063558 2021-11-20T08:57:12.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:12 smithi102 conmon[36472]: cluster 2021-11-20T08:57:11.528032+0000 mgr.smithi093.qdikkm (mgr.14162) 199 : cluster [DBG] 2021-11-20T08:57:12.928 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:12 smithi102 conmon[36472]: pgmap v162: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:12.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:12 smithi093 conmon[33522]: cluster 2021-11-20T08:57:11.528032+0000 mgr.smithi093.qdikkm (mgr.14162) 199 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:14 smithi102 conmon[36472]: cluster 2021-11-20T08:57:13.528646+0000 mgr.smithi093.qdikkm (mgr.14162) 200 : cluster [DBG] 2021-11-20T08:57:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:14 smithi102 conmon[36472]: pgmap v163: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:14.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:14 smithi093 conmon[33522]: cluster 2021-11-20T08:57:13.528646+0000 mgr.smithi093.qdikkm (mgr.14162) 200 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:16.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: cluster 2021-11-20T08:57:15.529137+0000 mgr.smithi093.qdikkm (mgr.14162) 201 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:16.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: cluster 2021-11-20T08:57: 2021-11-20T08:57:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: 16.558994+0000 mon.smithi093 (mon.0) 539 : cluster [WRN] Replacing daemon mds.cephfs.smithi102.lhrlmh as rank 0 with standby daemon mds.cephfs.smithi093.ljzugc 2021-11-20T08:57:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: cluster 2021-11-20T08:57:16.559069+0000 mon.smithi093 (mon.0) 540 : cluster [INF] MDS daemon mds.cephfs.smithi102.lhrlmh is removed because it is dead or otherwise unavailable. 2021-11-20T08:57:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: cluster 2021-11-20T08:57:16.559710+0000 mon.smithi093 (mon.0) 541 2021-11-20T08:57:16.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T08:57:16.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: cluster 2021-11-20T08:57:16.566140+0000 mon.smithi093 (mon. 2021-11-20T08:57:16.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: 0) 542 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T08:57:16.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:57:16.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: -11-20T08:57:16.571586+0000 mon.smithi093 (mon.0) 543 : cluster [DBG] 2021-11-20T08:57:16.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:16 smithi102 conmon[36472]: fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 2 up:standby 2021-11-20T08:57:16.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: cluster 2021-11-20T08:57:15.529137+0000 mgr.smithi093.qdikkm (mgr.14162) 201 2021-11-20T08:57:16.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:57:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: cluster 2021-11-20T08:57:16.558994+0000 mon.smithi093 (mon.0) 539 : cluster [WRN] Replacing daemon mds.cephfs.smithi102.lhrlmh as rank 0 with standby daemon mds.cephfs.smithi093.ljzugc 2021-11-20T08:57:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: cluster 2021-11-20T08:57:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: 2021-11-20T08:57:16.559069+0000 mon.smithi093 (mon.0) 540 : cluster [INF] MDS daemon mds.cephfs.smithi102.lhrlmh is removed because it is dead or otherwise unavailable. 2021-11-20T08:57:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:57:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: 20T08:57:16.559710+0000 mon.smithi093 ( 2021-11-20T08:57:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: mon.0) 541 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T08:57:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:57:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: -11-20T08:57:16.566140+0000 2021-11-20T08:57:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: mon.smithi093 (mon.0) 542 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T08:57:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: 2021-11-20T08:57:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: cluster 2021-11-20T08:57:16.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: 2021-11-20T08:57:16.571586+0000 mon.smithi093 2021-11-20T08:57:16.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:16 smithi093 conmon[33522]: (mon.0) 543 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 2 up:standby 2021-11-20T08:57:18.363 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:18.364+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:18.364 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:18.365+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:57:18.364 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:18.365+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:57:18.364 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:18.365+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:57:18.364 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:18.365+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 12.800000 2021-11-20T08:57:18.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:18 smithi102 conmon[36472]: cluster 2021-11-20T08:57:17.529502+0000 mgr.smithi093.qdikkm (mgr.14162) 202 : cluster [DBG] 2021-11-20T08:57:18.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:18 smithi102 conmon[36472]: pgmap v166: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:57:18.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:18 smithi102 conmon[36472]: cluster 2021-11-20T08:57:17.573563+0000 mon.smithi093 (mon.0) 544 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:reconnect 2021-11-20T08:57:18.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:18 smithi102 conmon[36472]: cluster 2021-11-20T08:57:17.573668+0000 mon.smithi093 (mon.0) 545 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 2 up:standby 2021-11-20T08:57:18.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:18 smithi102 conmon[36472]: cluster 2021-11-20T08:57:17.593050+0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:57:18.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: cluster 2021-11-20T08:57:17.529502+0000 mgr.smithi093.qdikkm 2021-11-20T08:57:18.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: (mgr.14162) 202 : cluster [DBG] pgmap v166: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:57:18.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: cluster 2021-11-20T08:57:17.573563+0000 mon.smithi093 (mon.0) 544 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:reconnect 2021-11-20T08:57:18.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: cluster 2021-11-20T08:57:17.573668+0000 mon.smithi093 (mon.0) 545 2021-11-20T08:57:18.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 2 up:standby 2021-11-20T08:57:18.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: cluster 2021-11-20T08:57: 2021-11-20T08:57:18.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: 17.593050+0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 : cluster 2021-11-20T08:57:18.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:18 smithi093 conmon[33522]: [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:57:19.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: cluster 2021-11-20T08:57:18.574951+0000 mon.smithi093 (mon.0) 546 : 2021-11-20T08:57:19.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:rejoin 2021-11-20T08:57:19.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: cluster 2021-11-20T08:57:18.575068+0000 mon.smithi093 (mon.0) 547 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 2 up:standby 2021-11-20T08:57:19.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: cluster 2021-11-20T08:57:19.575721+0000 mon.smithi093 (mon 2021-11-20T08:57:19.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: .0) 548 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:clientreplay 2021-11-20T08:57:19.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: cluster 2021-11-20T08:57:19.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: 2021-11-20T08:57:19.575777+0000 mon.smithi093 (mon. 2021-11-20T08:57:19.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: 0) 549 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:boot 2021-11-20T08:57:19.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:57:19.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: T08:57:19.575845+0000 mon.smithi093 (mon.0) 550 : cluster 2021-11-20T08:57:19.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:57:19.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T08:57:19.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: 20T08:57:19.575947+0000 mon.smithi093 (mon.0) 551 : audit [DBG] 2021-11-20T08:57:19.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:19 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T08:57:19.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: cluster 2021-11-20T08:57:18.574951+0000 mon.smithi093 (mon.0) 546 2021-11-20T08:57:19.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:rejoin 2021-11-20T08:57:19.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: cluster 2021-11-20T08:57:18.575068+0000 mon.smithi093 (mon.0) 547 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 2 up:standby 2021-11-20T08:57:19.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: cluster 2021-11-20T08:57:19.575721+0000 mon.smithi093 (mon.0) 548 : cluster [DBG] mds.? [v2:172.21.15.93:6826/2783208877,v1:172.21.15.93:6827/2783208877] up:clientreplay 2021-11-20T08:57:19.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: cluster 2021-11-20T08:57:19.575777+0000 mon.smithi093 (mon.0) 549 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:boot 2021-11-20T08:57:19.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: cluster 2021-11-20T08:57:19.575845+0000 mon.smithi093 (mon.0) 550 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:57:19.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: audit 2021-11-20T08:57:19.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:19 smithi093 conmon[33522]: 2021-11-20T08:57:19.575947+0000 mon.smithi093 (mon.0) 551 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T08:57:21.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:21 smithi102 conmon[36472]: cluster 2021-11-20T08:57:19.529924+0000 mgr.smithi093.qdikkm (mgr.14162) 203 : 2021-11-20T08:57:21.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:21 smithi102 conmon[36472]: cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:57:21.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:21 smithi093 conmon[33522]: cluster 2021-11-20T08:57:19.529924+0000 mgr.smithi093.qdikkm (mgr.14162) 203 : cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:57:23.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:23 smithi102 conmon[36472]: cluster 2021-11-20T08:57:21.530652+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:57:23.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:23 smithi102 conmon[36472]: ) 204 : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:57:23.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:23 smithi093 conmon[33522]: cluster 2021-11-20T08:57:21.530652+0000 mgr.smithi093.qdikkm (mgr.14162) 204 : cluster [DBG] 2021-11-20T08:57:23.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:23 smithi093 conmon[33522]: pgmap v168: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:57:24.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:24 smithi102 conmon[36472]: cluster 2021-11-20T08:57:23.318301+0000 mon.smithi093 (mon.0) 552 2021-11-20T08:57:24.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:24 smithi102 conmon[36472]: : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:standby 2021-11-20T08:57:24.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:24 smithi102 conmon[36472]: cluster 2021-11-20T08:57:23.318423+0000 mon.smithi093 (mon.0) 553 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:57:24.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:24 smithi093 conmon[33522]: cluster 2021-11-20T08:57:23.318301+0000 mon.smithi093 (mon.0) 552 : cluster [DBG] 2021-11-20T08:57:24.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:24 smithi093 conmon[33522]: mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:standby 2021-11-20T08:57:24.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:24 smithi093 conmon[33522]: cluster 2021-11-20T08:57:23.318423+0000 mon.smithi093 (mon.0) 553 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:57:25.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:25 smithi102 conmon[36472]: cluster 2021-11-20T08:57:23.531057+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:57:25.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:25 smithi102 conmon[36472]: .14162) 205 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:57:25.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:25 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:57:25.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:25 smithi093 conmon[33522]: 57:23.531057+0000 mgr.smithi093.qdikkm (mgr.14162) 205 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:57:27.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:27 smithi102 conmon[36472]: cluster 2021-11-20T08:57:25.531542+0000 mgr.smithi093.qdikkm (mgr.14162) 206 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:57:27.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:27 smithi093 conmon[33522]: cluster 2021-11-20T08:57:25.531542+0000 mgr.smithi093.qdikkm (mgr.14162) 206 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:57:29.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:29 smithi102 conmon[36472]: cluster 2021-11-20T08:57:27.531989+0000 mgr.smithi093.qdikkm (mgr.14162) 207 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 186 B/s wr, 1 op/s 2021-11-20T08:57:29.700 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:29 smithi093 conmon[33522]: cluster 2021-11-20T08:57:27.531989+0000 mgr.smithi093.qdikkm (mgr.14162) 207 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 186 B/s wr, 1 op/s 2021-11-20T08:57:29.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:29 smithi093 conmon[33522]: 2021-11-20T08:57:30.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:30 smithi102 conmon[36472]: cluster 2021-11-20T08:57:30.230211+0000 mon.smithi093 (mon.0) 554 : cluster [INF] 2021-11-20T08:57:30.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:30 smithi102 conmon[36472]: daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:57:30.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:30 smithi102 conmon[36472]: cluster 2021-11-20T08:57:30.234060+0000 mon.smithi093 (mon.0) 555 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T08:57:30.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:30 smithi093 conmon[33522]: debug 2021-11-20T08:57:30.332+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:57:30.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:30 smithi093 conmon[33522]: cluster 2021-11-20T08:57:30.230211+0000 2021-11-20T08:57:30.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:30 smithi093 conmon[33522]: mon.smithi093 (mon.0) 554 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:57:30.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:30 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:57:30.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:30 smithi093 conmon[33522]: 57:30.234060+0000 mon.smithi093 (mon.0) 555 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T08:57:31.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:31.173+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:31.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:31.173+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:57:31.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:31.173+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:57:31.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:31.173+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:57:31.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:31.173+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T08:57:31.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: cluster 2021-11-20T08:57:29.532451+0000 mgr.smithi093.qdikkm (mgr.14162) 208 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T08:57:31.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: cluster 2021-11-20T08:57:30.333663+0000 mon.smithi093 (mon.0) 556 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:57:31.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: cluster 2021-11-20T08:57:30.339252+0000 mon.smithi093 (mon.0) 557 : cluster 2021-11-20T08:57:31.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: [DBG] mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:boot 2021-11-20T08:57:31.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: cluster 2021-11-20T08:57: 2021-11-20T08:57:31.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: 30.339308+0000 mon.smithi093 (mon.0) 558 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:57:31.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:57:31.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: -11-20T08:57:30.339401+0000 2021-11-20T08:57:31.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: mon.smithi093 (mon.0) 559 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:57:31.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: 2021-11-20T08:57:31.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: cluster 2021-11-20T08:57:31.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: 2021-11-20T08:57:30.339598+0000 mon.smithi093 ( 2021-11-20T08:57:31.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: mon.0) 560 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:57:31.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: audit 2021-11 2021-11-20T08:57:31.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: -20T08:57:30.339706+0000 mon.smithi093 2021-11-20T08:57:31.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: (mon.0) 561 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:57:31.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:57:31.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: -11-20T08:57:30.344168 2021-11-20T08:57:31.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 562 : cluster [DBG] 2021-11-20T08:57:31.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:31 smithi102 conmon[36472]: fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:57:31.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: cluster 2021-11-20T08:57:29.532451+0000 mgr.smithi093.qdikkm (mgr.14162) 208 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T08:57:31.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: cluster 2021-11-20T08:57:30 2021-11-20T08:57:31.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: .333663+0000 mon.smithi093 (mon.0) 556 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:57:31.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: cluster 2021-11-20T08:57:30.339252+0000 mon.smithi093 (mon.0) 557 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:boot 2021-11-20T08:57:31.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: cluster 2021-11-20T08:57: 2021-11-20T08:57:31.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: 30.339308+0000 mon.smithi093 (mon.0) 558 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:57:31.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: cluster 2021-11-20T08:57:30. 2021-11-20T08:57:31.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: 339401+0000 mon.smithi093 (mon.0) 559 : cluster 2021-11-20T08:57:31.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:57:31.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: cluster 2021-11-20T08:57:30 2021-11-20T08:57:31.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: .339598+0000 mon.smithi093 (mon.0) 560 : cluster 2021-11-20T08:57:31.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:57:31.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: audit 2021-11-20T08:57 2021-11-20T08:57:31.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: :30.339706+0000 mon.smithi093 (mon.0) 561 2021-11-20T08:57:31.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:57:31.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T08:57:31.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: T08:57:30.344168+0000 mon.smithi093 (mon. 2021-11-20T08:57:31.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:31 smithi093 conmon[33522]: 0) 562 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:57:32.600 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:32 smithi093 conmon[33522]: cluster 2021-11-20T08:57:31.345160+0000 mon.smithi093 ( 2021-11-20T08:57:32.600 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:32 smithi093 conmon[33522]: mon.0) 563 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:reconnect 2021-11-20T08:57:32.601 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:32 smithi093 conmon[33522]: cluster 2021-11-20T08:57:31.345234+0000 mon.smithi093 (mon.0) 564 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:57:32.601 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:32 smithi093 conmon[33522]: cluster 2021-11-20T08:57:31.364895+0000 mds.cephfs.smithi093.zrryyo (mds 2021-11-20T08:57:32.601 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:32 smithi093 conmon[33522]: .0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:57:32.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:32 smithi102 conmon[36472]: cluster 2021-11-20T08:57:31.345160+0000 mon.smithi093 (mon.0) 563 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:reconnect 2021-11-20T08:57:32.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:32 smithi102 conmon[36472]: cluster 2021-11-20T08:57:31.345234+0000 mon.smithi093 (mon.0) 564 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:57:32.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:32 smithi102 conmon[36472]: cluster 2021-11-20T08:57:31.364895+0000 mds.cephfs.smithi093.zrryyo (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:57:33.453 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:33 smithi093 conmon[33522]: cluster 2021-11-20T08:57:31.532930 2021-11-20T08:57:33.454 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:33 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 209 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T08:57:33.454 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:33 smithi093 conmon[33522]: cluster 2021-11-20T08:57:32.349028+0000 mon.smithi093 (mon.0) 565 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:rejoin 2021-11-20T08:57:33.454 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:33 smithi093 conmon[33522]: cluster 2021-11-20T08:57:32.349071+0000 mon.smithi093 (mon 2021-11-20T08:57:33.455 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:33 smithi093 conmon[33522]: .0) 566 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:57:33.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:33 smithi102 conmon[36472]: cluster 2021-11-20T08:57:31.532930+0000 mgr.smithi093.qdikkm (mgr.14162) 209 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T08:57:33.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:33 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T08:57:33.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:33 smithi102 conmon[36472]: -20T08:57:32.349028+0000 mon.smithi093 (mon.0) 565 : cluster [DBG] mds.? [v2:172.21.15.93:6828/1976124814,v1:172.21.15.93:6829/1976124814] up:rejoin 2021-11-20T08:57:33.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:33 smithi102 conmon[36472]: cluster 2021-11-20T08:57:32.349071+0000 mon.smithi093 (mon.0) 566 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:57:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:35 smithi102 conmon[36472]: cluster 2021-11-20T08:57:33.533593+0000 mgr.smithi093.qdikkm (mgr.14162) 210 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:57:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:35 smithi102 conmon[36472]: 2021-11-20T08:57:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:35 smithi102 conmon[36472]: cluster 2021-11-20T08:57:34.356568+0000 mon.smithi093 (mon.0) 567 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:standby 2021-11-20T08:57:35.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:35 smithi102 conmon[36472]: cluster 2021-11-20T08:57:34.356638+0000 mon.smithi093 (mon.0) 568 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:57:35.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:35 smithi093 conmon[33522]: cluster 2021-11-20T08:57:33.533593+0000 mgr.smithi093.qdikkm (mgr.14162) 210 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:57:35.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:35 smithi093 conmon[33522]: cluster 2021-11-20T08:57:34.356568+0000 mon.smithi093 (mon.0) 567 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:standby 2021-11-20T08:57:35.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:35 smithi093 conmon[33522]: cluster 2021-11-20T08:57:34.356638+0000 mon.smithi093 (mon.0) 568 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:57:37.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:37 smithi102 conmon[36472]: cluster 2021-11-20T08:57:35.534048+0000 mgr.smithi093.qdikkm (mgr.14162) 211 : cluster [DBG] 2021-11-20T08:57:37.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:37 smithi102 conmon[36472]: pgmap v176: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:37 smithi093 conmon[33522]: cluster 2021-11-20T08:57:35.534048+0000 mgr.smithi093.qdikkm (mgr.14162) 211 : cluster [DBG] pgmap v176: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:39.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:39 smithi102 conmon[36472]: cluster 2021-11-20T08:57:37.534561+0000 mgr.smithi093.qdikkm (mgr.14162) 212 : cluster 2021-11-20T08:57:39.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:39 smithi102 conmon[36472]: [DBG] pgmap v177: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:39.700 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:39 smithi093 conmon[33522]: cluster 2021-11-20T08:57:37.534561+0000 mgr.smithi093.qdikkm (mgr.14162) 212 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:41.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:41 smithi093 conmon[33522]: cluster 2021-11-20T08:57:39.535023+0000 mgr.smithi093.qdikkm (mgr.14162) 213 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:41.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:41 smithi102 conmon[36472]: cluster 2021-11-20T08:57:39.535023+0000 mgr.smithi093.qdikkm (mgr.14162) 213 : 2021-11-20T08:57:41.713 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:41 smithi102 conmon[36472]: cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:43.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: debug 2021-11-20T08:57:43.367+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:57:43.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: cluster 2021-11-20T08:57:41. 2021-11-20T08:57:43.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: 535470+0000 mgr.smithi093.qdikkm (mgr.14162) 214 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:57:43.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: cluster 2021-11-20T08:57:43.207376+0000 2021-11-20T08:57:43.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: mon.smithi093 (mon.0) 569 : cluster [INF] daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:57:43.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: cluster 2021-11-20T08:57:43.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: 2021-11-20T08:57:43.213507+0000 2021-11-20T08:57:43.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:43 smithi093 conmon[33522]: mon.smithi093 (mon.0) 570 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T08:57:43.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:43 smithi102 conmon[36472]: cluster 2021-11-20T08:57:41.535470+0000 mgr.smithi093.qdikkm (mgr.14162) 214 : cluster [DBG] 2021-11-20T08:57:43.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:43 smithi102 conmon[36472]: pgmap v179: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:57:43.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:43 smithi102 conmon[36472]: cluster 2021-11-20T08:57:43.207376+0000 mon.smithi093 (mon.0) 569 : cluster [INF] daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:57:43.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:43 smithi102 conmon[36472]: cluster 2021-11-20T08:57 2021-11-20T08:57:43.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:43 smithi102 conmon[36472]: :43.213507+0000 mon.smithi093 (mon.0) 570 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T08:57:44.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: cluster 2021-11-20T08:57:43.368110+0000 mon.smithi093 (mon.0) 571 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:57:44.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: cluster 2021-11-20T08:57:43.371214 2021-11-20T08:57:44.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 572 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:boot 2021-11-20T08:57:44.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: cluster 2021-11-20T08:57:43.371261+0000 mon.smithi093 (mon.0) 573 : cluster [INF] Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:57:44.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: cluster 2021-11-20T08:57:43 2021-11-20T08:57:44.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: .371369+0000 mon.smithi093 (mon.0) 574 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:57:44.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: cluster 2021- 2021-11-20T08:57:44.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: 11-20T08:57:43.371638+0000 mon.smithi093 (mon.0) 575 : 2021-11-20T08:57:44.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:57:44.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: audit 2021-11-20T08:57 2021-11-20T08:57:44.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: :43.371735+0000 mon.smithi093 (mon.0) 576 : 2021-11-20T08:57:44.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:57:44.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:57:44.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: 57:43.375368+0000 mon.smithi093 (mon.0) 577 2021-11-20T08:57:44.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:44 smithi102 conmon[36472]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:57:44.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: cluster 2021-11-20T08:57:43.368110+0000 mon.smithi093 (mon 2021-11-20T08:57:44.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: .0) 571 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:57:44.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: cluster 2021-11-20T08:57:43.371214+0000 mon.smithi093 (mon.0) 572 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:boot 2021-11-20T08:57:44.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: cluster 2021-11-20T08:57:43.371261+0000 mon.smithi093 (mon.0) 573 : cluster [INF] 2021-11-20T08:57:44.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:57:44.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:57:44.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: 11-20T08:57:43.371369+0000 mon.smithi093 (mon.0) 574 : cluster [INF] 2021-11-20T08:57:44.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:57:44.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: cluster 2021-11-20T08:57:43 2021-11-20T08:57:44.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: .371638+0000 mon.smithi093 (mon.0) 575 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:57:44.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: 2021-11-20T08:57:44.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:57:44.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: 08:57:43.371735+0000 mon.smithi093 (mon.0) 2021-11-20T08:57:44.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: 576 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:57:44.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: cluster 2021-11-20T08:57:44.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: 2021-11-20T08:57:43.375368 2021-11-20T08:57:44.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 577 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:57:44.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:44 smithi093 conmon[33522]: 2021-11-20T08:57:45.585 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:45 smithi093 conmon[33522]: cluster 2021-11-20T08:57:43.535862+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:57:45.586 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:45 smithi093 conmon[33522]: ) 215 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:57:45.586 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:45 smithi093 conmon[33522]: cluster 2021-11-20T08:57:44.381327+0000 mon.smithi093 (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:reconnect 2021-11-20T08:57:45.586 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:45 smithi093 conmon[33522]: cluster 2021-11-20T08:57:45.587 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:45 smithi093 conmon[33522]: 2021-11-20T08:57:44.381415+0000 mon.smithi093 (mon.0) 579 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:57:45.588 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:45 smithi093 conmon[33522]: cluster 2021-11-20T08:57:44.400098+0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 2021-11-20T08:57:45.588 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:45 smithi093 conmon[33522]: : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:57:45.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: cluster 2021-11-20T08:57:43.535862+0000 mgr.smithi093.qdikkm (mgr.14162) 215 : cluster 2021-11-20T08:57:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: [DBG] pgmap v181: 65 pgs: 65 active+clean; 2.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:57:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: cluster 2021-11-20T08:57:44.381327+0000 mon.smithi093 (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:reconnect 2021-11-20T08:57:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T08:57:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: 08:57:44.381415+0000 mon.smithi093 (mon.0) 579 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:57:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: cluster 2021-11-20T08:57:44 2021-11-20T08:57:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: .400098+0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 : cluster 2021-11-20T08:57:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:45 smithi102 conmon[36472]: [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:57:46.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:46.188+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:57:46.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:46.189+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:57:46.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:46.189+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:57:46.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:46.189+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:57:46.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:57:46.189+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T08:57:46.444 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:46 smithi093 conmon[33522]: cluster 2021-11-20T08:57:45.381383+0000 mon.smithi093 (mon.0) 580 : cluster [DBG] 2021-11-20T08:57:46.444 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:46 smithi093 conmon[33522]: mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:rejoin 2021-11-20T08:57:46.445 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:46 smithi093 conmon[33522]: cluster 2021-11-20T08:57:45.381453+0000 mon.smithi093 (mon.0) 581 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:57:46.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:46 smithi102 conmon[36472]: cluster 2021-11-20T08:57:45.381383+0000 mon.smithi093 (mon.0) 580 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1149241327,v1:172.21.15.93:6827/1149241327] up:rejoin 2021-11-20T08:57:46.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:46 smithi102 conmon[36472]: cluster 2021-11-20T08:57:45.381453+0000 mon.smithi093 (mon 2021-11-20T08:57:46.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:46 smithi102 conmon[36472]: .0) 581 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:57:47.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:47 smithi102 conmon[36472]: cluster 2021-11-20T08:57:45.536383+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:57:47.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:47 smithi102 conmon[36472]: 216 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:57:47.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:47 smithi093 conmon[33522]: cluster 2021-11-20T08:57:45.536383+0000 mgr.smithi093.qdikkm (mgr.14162) 216 : cluster 2021-11-20T08:57:47.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:47 smithi093 conmon[33522]: [DBG] pgmap v182: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:57:48.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:48 smithi102 conmon[36472]: cluster 2021-11-20T08:57:47.536847+0000 2021-11-20T08:57:48.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:48 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 217 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:48.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:48 smithi102 conmon[36472]: cluster 2021-11-20T08:57:47.570429+0000 mon.smithi093 (mon.0) 582 : cluster [DBG] 2021-11-20T08:57:48.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:48 smithi102 conmon[36472]: mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:standby 2021-11-20T08:57:48.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:48 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T08:57:48.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:48 smithi102 conmon[36472]: 20T08:57:47.570498+0000 mon.smithi093 (mon.0 2021-11-20T08:57:48.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:48 smithi102 conmon[36472]: ) 583 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:57:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:48 smithi093 conmon[33522]: cluster 2021-11-20T08:57:47.536847+0000 mgr.smithi093.qdikkm (mgr.14162) 217 : cluster [DBG] 2021-11-20T08:57:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:48 smithi093 conmon[33522]: pgmap v183: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:48 smithi093 conmon[33522]: cluster 2021-11-20T08:57:47.570429+0000 mon.smithi093 (mon.0) 582 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:standby 2021-11-20T08:57:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:48 smithi093 conmon[33522]: cluster 2021-11-20T08:57:47.570498+0000 mon.smithi093 (mon.0) 583 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:57:50.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:50 smithi102 conmon[36472]: cluster 2021-11-20T08:57:49.537467+0000 mgr.smithi093.qdikkm (mgr.14162) 218 : cluster [DBG] pgmap v184: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:50 smithi093 conmon[33522]: cluster 2021-11-20T08:57:49.537467+0000 mgr.smithi093.qdikkm (mgr.14162) 218 : 2021-11-20T08:57:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:50 smithi093 conmon[33522]: cluster [DBG] pgmap v184: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:52.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:52 smithi102 conmon[36472]: cluster 2021-11-20T08:57:51.538021+0000 mgr.smithi093.qdikkm (mgr.14162) 219 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:52.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:52 smithi093 conmon[33522]: cluster 2021-11-20T08:57:51.538021+0000 mgr.smithi093.qdikkm (mgr.14162) 219 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:52.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:52 smithi093 conmon[33522]: 2021-11-20T08:57:54.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:54 smithi102 conmon[36472]: cluster 2021-11-20T08:57:53.538540+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:57:54.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:54 smithi102 conmon[36472]: 220 : cluster [DBG] pgmap v186: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:54.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:54 smithi093 conmon[33522]: cluster 2021-11-20T08:57:53.538540+0000 mgr.smithi093.qdikkm (mgr.14162) 220 : cluster 2021-11-20T08:57:54.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:54 smithi093 conmon[33522]: [DBG] pgmap v186: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:57:56.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:56 smithi102 conmon[36472]: cluster 2021-11-20T08:57:55.539143+0000 mgr.smithi093.qdikkm (mgr.14162) 221 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2021-11-20T08:57:56.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:56 smithi093 conmon[33522]: cluster 2021-11-20T08:57:55.539143+0000 mgr.smithi093.qdikkm (mgr.14162) 221 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2021-11-20T08:57:57.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:57 smithi102 conmon[36472]: audit 2021-11-20T08:57:56.621266+0000 mon.smithi093 (mon.0) 584 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:57:57.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:57 smithi102 conmon[36472]: audit 2021-11-20T08:57:56.622232+0000 mon.smithi093 (mon.0) 585 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:57:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:57 smithi093 conmon[33522]: audit 2021-11-20T08:57:56.621266+0000 mon.smithi093 (mon.0) 584 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:57:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:57 smithi093 conmon[33522]: audit 2021-11-20T08:57:56.622232 2021-11-20T08:57:57.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:57 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 585 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:57:58.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:57:58 smithi102 conmon[36472]: cluster 2021-11-20T08:57:57.539715+0000 mgr.smithi093.qdikkm (mgr.14162) 222 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T08:57:58.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:58 smithi093 conmon[33522]: cluster 2021-11-20T08:57:57.539715+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:57:58.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:57:58 smithi093 conmon[33522]: ) 222 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T08:58:00.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:00 smithi102 conmon[36472]: cluster 2021-11-20T08:57:59.540279+0000 mgr.smithi093.qdikkm (mgr.14162) 223 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:58:00.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:00 smithi102 conmon[36472]: cluster 2021-11-20T08:58:00.206292+0000 2021-11-20T08:58:00.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:00 smithi102 conmon[36472]: mon.smithi093 (mon.0) 586 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:58:00.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:00 smithi102 conmon[36472]: cluster 2021-11-20T08:58:00.211889+0000 mon.smithi093 (mon.0) 587 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T08:58:00.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: debug 2021-11-20T08:58:00.593+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:58:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: -11-20T08:57:59.540279+0000 mgr.smithi093.qdikkm (mgr.14162) 223 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:58:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:58:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: -11-20T08:58:00.206292+0000 mon.smithi093 (mon.0) 586 : cluster 2021-11-20T08:58:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:58:00.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: cluster 2021-11-20T08:58:00.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: 2021-11-20T08:58:00.211889+0000 mon.smithi093 (mon.0) 587 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T08:58:00.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:00 smithi093 conmon[33522]: 2021-11-20T08:58:01.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:01.198+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:58:01.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:01.199+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:58:01.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:01.199+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:58:01.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:01.199+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:58:01.200 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:01.199+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T08:58:01.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: cluster 2021-11-20T08:58:00.595516+0000 mon.smithi093 (mon.0) 588 : cluster 2021-11-20T08:58:01.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:01.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: cluster 2021-11-20T08:58:00.603991+0000 mon.smithi093 (mon.0) 589 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:boot 2021-11-20T08:58:01.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: cluster 2021-11-20T08:58:00.604051+0000 mon.smithi093 (mon.0) 2021-11-20T08:58:01.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: 590 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:58:01.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:58:01.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: 58:00.604164+0000 mon.smithi093 (mon.0) 591 : cluster 2021-11-20T08:58:01.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:01.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: cluster 2021-11-20T08:58 2021-11-20T08:58:01.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: :00.604505+0000 mon.smithi093 (mon.0) 592 : cluster 2021-11-20T08:58:01.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:01.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: audit 2021-11-20T08:58 2021-11-20T08:58:01.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: :00.604641+0000 mon.smithi093 (mon.0) 593 : audit 2021-11-20T08:58:01.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:58:01.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: cluster 2021-11-20T08:58:00.609832+0000 mon.smithi093 2021-11-20T08:58:01.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:01 smithi102 conmon[36472]: (mon.0) 594 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:58:01.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: cluster 2021-11-20T08:58:00.595516+0000 mon.smithi093 (mon.0) 588 : cluster [ERR] 2021-11-20T08:58:01.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:01.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: cluster 2021-11-20T08:58:00.603991+0000 mon.smithi093 (mon.0) 589 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:boot 2021-11-20T08:58:01.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: cluster 2021-11-20T08:58:00.604051+0000 mon.smithi093 (mon.0 2021-11-20T08:58:01.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: ) 590 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:58:01.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: cluster 2021-11-20T08:58:00.604164+0000 2021-11-20T08:58:01.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: mon.smithi093 (mon.0) 591 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:01.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:58:01.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: 08:58:00.604505+0000 mon.smithi093 (mon.0 2021-11-20T08:58:01.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: ) 592 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:01.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: audit 2021-11-20 2021-11-20T08:58:01.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: T08:58:00.604641+0000 mon.smithi093 (mon.0 2021-11-20T08:58:01.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: ) 593 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:58:01.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:58:01.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: 08:58:00.609832+0000 mon.smithi093 (mon.0) 2021-11-20T08:58:01.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:01 smithi093 conmon[33522]: 594 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:58:02.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:02 smithi102 conmon[36472]: cluster 2021-11-20T08:58:01.540928+0000 mgr.smithi093.qdikkm (mgr.14162) 224 : 2021-11-20T08:58:02.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:02 smithi102 conmon[36472]: cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:58:02.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:02 smithi102 conmon[36472]: cluster 2021-11-20T08:58:01.617472+0000 mon.smithi093 (mon.0) 595 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:reconnect 2021-11-20T08:58:02.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:02 smithi102 conmon[36472]: cluster 2021-11-20T08:58:01.617535+0000 mon.smithi093 (mon.0) 596 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:58:02.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:02 smithi102 conmon[36472]: cluster 2021-11-20T08:58:01.637526+0000 mds.cephfs.smithi093.zrryyo (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:58:02.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:02 smithi093 conmon[33522]: cluster 2021-11-20T08:58:01.540928+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:58:02.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:02 smithi093 conmon[33522]: 224 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:58:02.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:02 smithi093 conmon[33522]: cluster 2021-11-20T08:58:01.617472+0000 mon.smithi093 (mon.0) 595 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:reconnect 2021-11-20T08:58:02.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:02 smithi093 conmon[33522]: cluster 2021-11-20T08:58:01.617535+0000 mon.smithi093 (mon.0) 596 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:58:02.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:02 smithi093 conmon[33522]: cluster 2021-11-20T08:58:01.637526+0000 mds.cephfs.smithi093.zrryyo (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0219995 2021-11-20T08:58:03.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: cluster 2021-11-20T08:58:02.619863+0000 mon.smithi093 ( 2021-11-20T08:58:03.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: mon.0) 597 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:rejoin 2021-11-20T08:58:03.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: cluster 2021-11-20T08:58:02.619935+0000 mon.smithi093 (mon.0) 598 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:58:03.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: audit 2021-11-20T08:58:02.788438+0000 mon.smithi093 (mon.0) 599 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:58:03.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: audit 2021-11-20T08:58:03.168611+0000 mon.smithi093 (mon.0) 600 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:58:03.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: cluster 2021-11-20T08:58:03.620624+0000 mon.smithi093 (mon.0) 601 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:clientreplay 2021-11-20T08:58:03.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:58:03.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: 58:03.620674+0000 mon.smithi093 (mon.0) 602 2021-11-20T08:58:03.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:03 smithi102 conmon[36472]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:58:03.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: cluster 2021-11-20T08:58:02.619863+0000 mon.smithi093 ( 2021-11-20T08:58:03.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: mon.0) 597 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:rejoin 2021-11-20T08:58:03.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: cluster 2021-11-20T08:58:02.619935+0000 mon.smithi093 (mon.0) 598 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:58:03.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: audit 2021-11-20T08:58:02.788438+0000 mon.smithi093 (mon.0) 599 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:58:03.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: audit 2021-11-20T08:58:03.168611+0000 mon.smithi093 (mon.0) 600 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:58:03.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: cluster 2021-11-20T08:58:03.620624+0000 mon.smithi093 (mon.0) 601 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2013720610,v1:172.21.15.93:6829/2013720610] up:clientreplay 2021-11-20T08:58:03.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:58:03.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:03 smithi093 conmon[33522]: 08:58:03.620674+0000 mon.smithi093 (mon.0) 602 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:58:05.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:05 smithi093 conmon[33522]: cluster 2021-11-20T08:58:03.541392+0000 mgr.smithi093.qdikkm (mgr.14162) 225 : cluster [DBG] 2021-11-20T08:58:05.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:05 smithi093 conmon[33522]: pgmap v192: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:05.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:05 smithi102 conmon[36472]: cluster 2021-11-20T08:58:03.541392+0000 mgr.smithi093.qdikkm (mgr.14162) 225 : 2021-11-20T08:58:05.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:05 smithi102 conmon[36472]: cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:06.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:06 smithi093 conmon[33522]: cluster 2021-11-20T08:58:05.172391+0000 mon.smithi093 (mon.0) 603 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:standby 2021-11-20T08:58:06.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:06 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:58:06.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:06 smithi093 conmon[33522]: 08:58:05.172453+0000 mon.smithi093 (mon.0) 604 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:58:06.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:06 smithi102 conmon[36472]: cluster 2021-11-20T08:58:05.172391+0000 mon.smithi093 2021-11-20T08:58:06.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:06 smithi102 conmon[36472]: (mon.0) 603 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:standby 2021-11-20T08:58:06.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:06 smithi102 conmon[36472]: cluster 2021-11-20T08:58:05.172453+0000 mon.smithi093 2021-11-20T08:58:06.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:06 smithi102 conmon[36472]: (mon.0) 604 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:58:07.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:07 smithi093 conmon[33522]: cluster 2021-11-20T08:58:05.541866+0000 mgr.smithi093.qdikkm (mgr.14162) 226 : cluster 2021-11-20T08:58:07.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:07 smithi093 conmon[33522]: [DBG] pgmap v193: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:07.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:07 smithi102 conmon[36472]: cluster 2021-11-20T08:58:05.541866+0000 mgr.smithi093.qdikkm (mgr.14162) 226 : cluster 2021-11-20T08:58:07.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:07 smithi102 conmon[36472]: [DBG] pgmap v193: 65 pgs: 65 active+clean; 2.6 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:09.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:09 smithi093 conmon[33522]: cluster 2021-11-20T08:58:07.542289+0000 mgr.smithi093.qdikkm (mgr.14162) 227 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:58:09.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:09 smithi102 conmon[36472]: cluster 2021-11-20T08:58:07.542289+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:58:09.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:09 smithi102 conmon[36472]: ) 227 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:58:11.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:11 smithi093 conmon[33522]: cluster 2021-11-20T08:58:09.542827+0000 mgr.smithi093.qdikkm (mgr.14162) 228 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:58:11.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:11 smithi102 conmon[36472]: cluster 2021-11-20T08:58:09.542827+0000 mgr.smithi093.qdikkm (mgr.14162) 228 : cluster 2021-11-20T08:58:11.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:11 smithi102 conmon[36472]: [DBG] pgmap v195: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:58:13.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:13 smithi093 conmon[33522]: cluster 2021-11-20T08:58:11.543311+0000 mgr.smithi093.qdikkm (mgr.14162) 229 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 180 B/s wr, 1 op/s 2021-11-20T08:58:13.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:13 smithi102 conmon[36472]: cluster 2021-11-20T08:58:11.543311+0000 mgr.smithi093.qdikkm (mgr.14162) 229 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 180 B/s wr, 1 op/s 2021-11-20T08:58:15.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:15 smithi093 conmon[33522]: cluster 2021-11-20T08:58:13.544008+0000 mgr.smithi093.qdikkm (mgr.14162) 230 : cluster 2021-11-20T08:58:15.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:15 smithi093 conmon[33522]: [DBG] pgmap v197: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:58:15.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:15 smithi102 conmon[36472]: cluster 2021-11-20T08:58:13.544008+0000 mgr.smithi093.qdikkm (mgr.14162) 230 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:58:16.213 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:16.214+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:58:16.214 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:16.215+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:58:16.214 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:16.215+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:58:16.214 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:16.215+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:58:16.214 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:16.215+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T08:58:17.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:17 smithi093 conmon[33522]: cluster 2021-11-20T08:58:15.544456+0000 mgr.smithi093.qdikkm (mgr.14162) 231 : cluster 2021-11-20T08:58:17.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:17 smithi093 conmon[33522]: [DBG] pgmap v198: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:58:17.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:17 smithi102 conmon[36472]: cluster 2021-11-20T08:58:15.544456+0000 mgr.smithi093.qdikkm (mgr.14162) 231 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:58:19.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:19 smithi093 conmon[33522]: debug 2021-11-20T08:58:19.198+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:19.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:19 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:58:19.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:19 smithi093 conmon[33522]: 58:17.544978+0000 mgr.smithi093.qdikkm (mgr.14162) 232 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:58:19.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:19 smithi093 conmon[33522]: cluster 2021-11-20T08:58:18.210300+0000 mon.smithi093 (mon.0) 605 : 2021-11-20T08:58:19.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:19 smithi093 conmon[33522]: cluster [INF] daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:58:19.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:19 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:58:19.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:19 smithi093 conmon[33522]: -11-20T08:58:18.214997+0000 mon.smithi093 (mon.0) 606 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2021-11-20T08:58:19.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:19 smithi102 conmon[36472]: cluster 2021-11-20T08:58:17.544978+0000 mgr.smithi093.qdikkm (mgr.14162) 232 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:58:19.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:19 smithi102 conmon[36472]: cluster 2021-11-20T08:58:18.210300+0000 mon.smithi093 (mon.0) 605 : cluster [INF] daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:58:19.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:19 smithi102 conmon[36472]: cluster 2021-11-20T08:58:18.214997+0000 mon.smithi093 (mon.0) 606 : cluster [DBG] 2021-11-20T08:58:19.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:19 smithi102 conmon[36472]: osdmap e48: 6 total, 6 up, 6 in 2021-11-20T08:58:20.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: cluster 2021-11-20T08:58:19.199099+0000 mon.smithi093 (mon.0) 607 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:20.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: cluster 2021-11-20T08:58: 2021-11-20T08:58:20.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: 19.207610+0000 mon.smithi093 (mon.0) 608 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:boot 2021-11-20T08:58:20.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: cluster 2021-11-20T08:58:19.207684+0000 mon.smithi093 (mon.0) 609 : cluster [INF] Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:58:20.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:58:20.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: 08:58:19.207787+0000 mon.smithi093 (mon.0) 610 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:20.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: cluster 2021-11-20T08 2021-11-20T08:58:20.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: :58:19.208016+0000 mon.smithi093 (mon.0 2021-11-20T08:58:20.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: ) 611 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:20.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:58:20.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: 08:58:19.208148+0000 mon.smithi093 (mon. 2021-11-20T08:58:20.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: 0) 612 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:58:20.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:58:20.466 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: 11-20T08:58:19.213380+0000 mon.smithi093 2021-11-20T08:58:20.466 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:20 smithi093 conmon[33522]: (mon.0) 613 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:58:20.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: cluster 2021-11-20T08:58:19.199099+0000 mon.smithi093 (mon.0) 607 : cluster [ERR] 2021-11-20T08:58:20.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:20.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: cluster 2021-11-20T08:58:19.207610+0000 mon.smithi093 (mon.0) 608 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:boot 2021-11-20T08:58:20.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: cluster 2021-11-20T08:58:19.207684+0000 mon.smithi093 (mon. 2021-11-20T08:58:20.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: 0) 609 : cluster [INF] Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:58:20.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: cluster 2021-11-20T08:58:19.207787+0000 mon.smithi093 (mon.0) 610 : cluster 2021-11-20T08:58:20.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:20.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: cluster 2021-11-20T08:58:19.208016+0000 mon.smithi093 (mon.0) 611 : cluster [DBG] 2021-11-20T08:58:20.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:20.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: audit 2021-11-20T08:58:19.208148+0000 2021-11-20T08:58:20.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: mon.smithi093 (mon.0) 612 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:58:20.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: cluster 2021- 2021-11-20T08:58:20.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:20 smithi102 conmon[36472]: 11-20T08:58:19.213380+0000 mon.smithi093 (mon.0) 613 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:58:21.452 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:21 smithi093 conmon[33522]: cluster 2021-11-20T08:58:19.545383+0000 mgr.smithi093.qdikkm (mgr.14162) 233 : cluster [DBG] 2021-11-20T08:58:21.452 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:21 smithi093 conmon[33522]: pgmap v201: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:58:21.452 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:21 smithi093 conmon[33522]: cluster 2021-11-20T08:58:20.216626+0000 mon.smithi093 (mon.0) 614 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:reconnect 2021-11-20T08:58:21.452 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:21 smithi093 conmon[33522]: cluster 2021-11-20T08:58:20.216692+0000 mon.smithi093 (mon.0) 615 : 2021-11-20T08:58:21.453 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:21 smithi093 conmon[33522]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:58:21.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:21 smithi102 conmon[36472]: cluster 2021-11-20T08:58:19.545383+0000 mgr.smithi093.qdikkm (mgr.14162) 233 : cluster [DBG] 2021-11-20T08:58:21.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:21 smithi102 conmon[36472]: pgmap v201: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:58:21.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:21 smithi102 conmon[36472]: cluster 2021-11-20T08:58:20.216626+0000 mon.smithi093 (mon.0) 614 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:reconnect 2021-11-20T08:58:21.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:21 smithi102 conmon[36472]: cluster 2021-11-20T08:58:20.216692+0000 mon.smithi093 (mon.0 2021-11-20T08:58:21.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:21 smithi102 conmon[36472]: ) 615 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:58:22.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:22 smithi102 conmon[36472]: cluster 2021-11-20T08:58:21.218704+0000 mon.smithi093 (mon.0) 616 : cluster [DBG] 2021-11-20T08:58:22.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:22 smithi102 conmon[36472]: mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:rejoin 2021-11-20T08:58:22.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:22 smithi102 conmon[36472]: cluster 2021-11-20T08:58:21.218765+0000 mon.smithi093 (mon.0) 617 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:22.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:22 smithi093 conmon[33522]: cluster 2021-11-20T08:58:21.218704+0000 mon.smithi093 (mon.0) 616 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3729373254,v1:172.21.15.93:6827/3729373254] up:rejoin 2021-11-20T08:58:22.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:22 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T08:58:22.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:22 smithi093 conmon[33522]: T08:58:21.218765+0000 mon.smithi093 (mon.0) 617 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:23.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:23 smithi093 conmon[33522]: cluster 2021-11-20T08:58:21.545912+0000 mgr.smithi093.qdikkm (mgr.14162) 234 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:58:23.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:23 smithi093 conmon[33522]: 2021-11-20T08:58:23.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:23 smithi093 conmon[33522]: cluster 2021-11-20T08:58:22.576738+0000 mon.smithi093 (mon.0) 618 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:standby 2021-11-20T08:58:23.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:23 smithi093 conmon[33522]: cluster 2021-11-20T08:58:22.576813+0000 mon.smithi093 (mon.0) 619 : cluster 2021-11-20T08:58:23.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:23 smithi093 conmon[33522]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:23.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:23 smithi102 conmon[36472]: cluster 2021-11-20T08:58:21.545912+0000 mgr.smithi093.qdikkm (mgr.14162) 234 : 2021-11-20T08:58:23.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:23 smithi102 conmon[36472]: cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:58:23.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:23 smithi102 conmon[36472]: cluster 2021-11-20T08:58:22.576738+0000 mon.smithi093 (mon.0) 618 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:standby 2021-11-20T08:58:23.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:23 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:58:23.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:23 smithi102 conmon[36472]: 58:22.576813+0000 mon.smithi093 (mon.0) 619 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:25.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:25 smithi102 conmon[36472]: cluster 2021-11-20T08:58:23.546423+0000 mgr.smithi093.qdikkm (mgr.14162) 235 : cluster 2021-11-20T08:58:25.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:25 smithi102 conmon[36472]: [DBG] pgmap v203: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:25.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:25 smithi093 conmon[33522]: cluster 2021-11-20T08:58:23.546423+0000 mgr.smithi093.qdikkm (mgr.14162) 235 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:27.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:27 smithi102 conmon[36472]: cluster 2021-11-20T08:58:25.546853+0000 mgr.smithi093.qdikkm (mgr.14162) 236 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:27.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:27 smithi093 conmon[33522]: cluster 2021-11-20T08:58:25.546853+0000 mgr.smithi093.qdikkm (mgr.14162) 236 : 2021-11-20T08:58:27.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:27 smithi093 conmon[33522]: cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:29.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:29 smithi093 conmon[33522]: cluster 2021-11-20T08:58:27.547359+0000 mgr.smithi093.qdikkm (mgr.14162) 237 : cluster [DBG] 2021-11-20T08:58:29.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:29 smithi093 conmon[33522]: pgmap v205: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:29.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:29 smithi102 conmon[36472]: cluster 2021-11-20T08:58:27.547359+0000 mgr.smithi093.qdikkm (mgr.14162) 237 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:29.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:29 smithi102 conmon[36472]: 2021-11-20T08:58:31.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:31.230+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:58:31.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:31.231+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:58:31.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:31.231+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:58:31.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:31.231+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:58:31.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:31.231+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T08:58:31.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:31 smithi093 conmon[33522]: debug 2021-11-20T08:58:31.237+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:31.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:31 smithi093 conmon[33522]: cluster 2021-11-20T08:58:29.547846+0000 mgr.smithi093.qdikkm 2021-11-20T08:58:31.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:31 smithi093 conmon[33522]: (mgr.14162) 238 : cluster [DBG] pgmap v206: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:31.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:31 smithi093 conmon[33522]: cluster 2021-11-20T08:58:31.192281+0000 mon.smithi093 (mon.0) 620 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:58:31.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:31 smithi093 conmon[33522]: cluster 2021-11-20T08:58:31.196378+0000 mon.smithi093 (mon. 2021-11-20T08:58:31.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:31 smithi093 conmon[33522]: 0) 621 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-20T08:58:31.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:31 smithi102 conmon[36472]: cluster 2021-11-20T08:58:29.547846+0000 mgr.smithi093.qdikkm (mgr.14162) 238 : cluster [DBG] pgmap v206: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:31.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:31 smithi102 conmon[36472]: cluster 2021-11-20T08:58: 2021-11-20T08:58:31.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:31 smithi102 conmon[36472]: 31.192281+0000 mon.smithi093 (mon.0) 620 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:58:31.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:31 smithi102 conmon[36472]: cluster 2021-11-20T08:58:31.196378+0000 mon.smithi093 (mon.0) 621 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-20T08:58:32.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: cluster 2021-11-20T08:58:31.238390+0000 mon.smithi093 (mon 2021-11-20T08:58:32.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: .0) 622 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:32.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: cluster 2021-11-20T08:58:31.259974+0000 mon.smithi093 (mon.0) 623 : cluster [DBG] 2021-11-20T08:58:32.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:boot 2021-11-20T08:58:32.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T08:58:32.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: -20T08:58:31.260019+0000 mon.smithi093 ( 2021-11-20T08:58:32.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: mon.0) 624 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:58:32.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: cluster 2021-11-20T08:58:32.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: 2021-11-20T08:58: 2021-11-20T08:58:32.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: 31.260107+0000 mon.smithi093 (mon.0 2021-11-20T08:58:32.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: ) 625 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:32.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T08:58:32.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: -20T08:58:31.260342+0000 mon.smithi093 (mon. 2021-11-20T08:58:32.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: 0) 626 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:32.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: audit 2021-11-20T08:58:32.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: 2021-11-20T08:58: 2021-11-20T08:58:32.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: 31.260791+0000 mon.smithi093 (mon.0) 627 : 2021-11-20T08:58:32.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:58:32.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T08:58:32.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: T08:58:31.263749+0000 2021-11-20T08:58:32.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:32 smithi102 conmon[36472]: mon.smithi093 (mon.0) 628 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:58:32.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: cluster 2021-11-20T08:58:31.238390+0000 mon.smithi093 (mon.0) 622 : 2021-11-20T08:58:32.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:32.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: cluster 2021-11-20T08:58:31.259974+0000 mon.smithi093 (mon.0) 623 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:boot 2021-11-20T08:58:32.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: cluster 2021-11-20T08:58:31.260019+0000 mon.smithi093 (mon.0) 624 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:58:32.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: cluster 2021-11-20T08:58 2021-11-20T08:58:32.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: :31.260107+0000 mon.smithi093 (mon.0) 625 : cluster [INF] 2021-11-20T08:58:32.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:32.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: cluster 2021-11-20T08:58:31 2021-11-20T08:58:32.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: .260342+0000 mon.smithi093 (mon.0) 626 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:32.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: 2021-11-20T08:58:32.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: audit 2021-11-20T08:58:32.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: 2021-11-20T08:58:31.260791+0000 mon.smithi093 (mon.0) 627 : audit 2021-11-20T08:58:32.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:58:32.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:58:32.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: 08:58:31.263749+0000 mon.smithi093 (mon 2021-11-20T08:58:32.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: .0) 628 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:58:32.719 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:32 smithi093 conmon[33522]: 2021-11-20T08:58:33.595 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:33 smithi093 conmon[33522]: cluster 2021-11-20T08:58:31.548227+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:58:33.595 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:33 smithi093 conmon[33522]: .14162) 239 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:33.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:33 smithi093 conmon[33522]: cluster 2021-11-20T08:58:32.269160+0000 mon.smithi093 (mon.0) 629 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:reconnect 2021-11-20T08:58:33.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:33 smithi093 conmon[33522]: cluster 2021-11-20T08:58:32.269225+0000 mon.smithi093 (mon.0 2021-11-20T08:58:33.596 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:33 smithi093 conmon[33522]: ) 630 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:58:33.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:33 smithi093 conmon[33522]: cluster 2021-11-20T08:58:32. 2021-11-20T08:58:33.597 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:33 smithi093 conmon[33522]: 287779+0000 mds.cephfs.smithi093.zrryyo (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:58:33.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: cluster 2021-11-20T08:58:31.548227+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:58:33.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: 239 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:33.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: cluster 2021-11-20T08:58:32.269160+0000 mon.smithi093 (mon.0) 629 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:reconnect 2021-11-20T08:58:33.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:58:33.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: 58:32.269225+0000 mon.smithi093 (mon. 2021-11-20T08:58:33.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: 0) 630 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:58:33.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: cluster 2021-11-20T08:58:33.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: 2021-11-20T08:58: 2021-11-20T08:58:33.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: 32.287779+0000 mds.cephfs.smithi093.zrryyo (mds.0) 1 2021-11-20T08:58:33.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:33 smithi102 conmon[36472]: : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:58:34.444 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:34 smithi093 conmon[33522]: cluster 2021-11-20T08:58:33.270033+0000 2021-11-20T08:58:34.444 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:34 smithi093 conmon[33522]: mon.smithi093 (mon.0) 631 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:rejoin 2021-11-20T08:58:34.445 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:34 smithi093 conmon[33522]: cluster 2021-11-20T08:58:33.270100+0000 2021-11-20T08:58:34.445 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:34 smithi093 conmon[33522]: mon.smithi093 (mon.0) 632 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:58:34.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:34 smithi102 conmon[36472]: cluster 2021-11-20T08:58:33.270033 2021-11-20T08:58:34.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:34 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 631 : cluster [DBG] mds.? [v2:172.21.15.93:6828/4247980838,v1:172.21.15.93:6829/4247980838] up:rejoin 2021-11-20T08:58:34.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:34 smithi102 conmon[36472]: cluster 2021-11-20T08:58:33.270100+0000 mon.smithi093 ( 2021-11-20T08:58:34.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:34 smithi102 conmon[36472]: mon.0) 632 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:58:35.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:35 smithi102 conmon[36472]: cluster 2021-11-20T08:58:33.548616+0000 mgr.smithi093.qdikkm (mgr.14162) 240 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:35.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:35 smithi102 conmon[36472]: 2021-11-20T08:58:35.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:35 smithi093 conmon[33522]: cluster 2021-11-20T08:58:33.548616+0000 mgr.smithi093.qdikkm (mgr.14162) 240 : cluster 2021-11-20T08:58:35.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:35 smithi093 conmon[33522]: [DBG] pgmap v209: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:36.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:36 smithi102 conmon[36472]: cluster 2021-11-20T08:58:35.284105+0000 mon.smithi093 (mon 2021-11-20T08:58:36.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:36 smithi102 conmon[36472]: .0) 633 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:standby 2021-11-20T08:58:36.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:36 smithi102 conmon[36472]: cluster 2021-11-20T08:58:35.284168+0000 mon.smithi093 (mon.0) 634 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:58:36.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:36 smithi093 conmon[33522]: cluster 2021-11-20T08:58:35.284105+0000 mon.smithi093 (mon.0) 633 : cluster 2021-11-20T08:58:36.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:36 smithi093 conmon[33522]: [DBG] mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:standby 2021-11-20T08:58:36.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:36 smithi093 conmon[33522]: cluster 2021-11-20T08:58:35.284168+0000 mon.smithi093 (mon.0) 634 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:58:37.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:37 smithi102 conmon[36472]: cluster 2021-11-20T08:58:35.549082+0000 mgr.smithi093.qdikkm (mgr.14162) 241 : cluster [DBG] 2021-11-20T08:58:37.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:37 smithi102 conmon[36472]: pgmap v210: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:37 smithi093 conmon[33522]: cluster 2021-11-20T08:58:35.549082+0000 mgr.smithi093.qdikkm (mgr.14162) 241 : cluster 2021-11-20T08:58:37.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:37 smithi093 conmon[33522]: [DBG] pgmap v210: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:39.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:39 smithi102 conmon[36472]: cluster 2021-11-20T08:58:37.549595+0000 mgr.smithi093.qdikkm (mgr.14162) 242 : cluster 2021-11-20T08:58:39.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:39 smithi102 conmon[36472]: [DBG] pgmap v211: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:39.700 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:39 smithi093 conmon[33522]: cluster 2021-11-20T08:58:37.549595+0000 mgr.smithi093.qdikkm (mgr.14162) 242 : cluster 2021-11-20T08:58:39.702 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:39 smithi093 conmon[33522]: [DBG] pgmap v211: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:41.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:41 smithi102 conmon[36472]: cluster 2021-11-20T08:58:39.550086+0000 mgr.smithi093.qdikkm (mgr.14162) 243 : cluster 2021-11-20T08:58:41.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:41 smithi102 conmon[36472]: [DBG] pgmap v212: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:41.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:41 smithi093 conmon[33522]: cluster 2021-11-20T08:58:39.550086+0000 mgr.smithi093.qdikkm (mgr.14162) 243 : cluster [DBG] 2021-11-20T08:58:41.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:41 smithi093 conmon[33522]: pgmap v212: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:43.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:43 smithi102 conmon[36472]: cluster 2021-11-20T08:58:41.550647+0000 mgr.smithi093.qdikkm (mgr.14162) 244 : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:43.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:43 smithi093 conmon[33522]: cluster 2021-11-20T08:58:41.550647+0000 mgr.smithi093.qdikkm (mgr.14162) 244 : cluster [DBG] 2021-11-20T08:58:43.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:43 smithi093 conmon[33522]: pgmap v213: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:44.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:44 smithi102 conmon[36472]: cluster 2021-11-20T08:58:44.188686+0000 mon.smithi093 (mon.0) 635 : cluster [INF] daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:58:44.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:44 smithi102 conmon[36472]: cluster 2021-11-20T08:58:44.192391+0000 mon.smithi093 (mon.0) 636 : 2021-11-20T08:58:44.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:44 smithi102 conmon[36472]: cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-20T08:58:44.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:44 smithi093 conmon[33522]: debug 2021-11-20T08:58:44.301+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:44.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:44 smithi093 conmon[33522]: cluster 2021-11-20T08:58 2021-11-20T08:58:44.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:44 smithi093 conmon[33522]: :44.188686+0000 mon.smithi093 (mon.0) 635 : cluster [INF] daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:58:44.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:44 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:58:44.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:44 smithi093 conmon[33522]: 20T08:58:44.192391+0000 mon.smithi093 (mon.0) 636 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-20T08:58:45.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster 2021-11-20T08:58:43.551293+0000 mgr.smithi093.qdikkm (mgr.14162) 245 : 2021-11-20T08:58:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T08:58:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster 2021-11-20T08:58:44.303766+0000 mon.smithi093 (mon.0) 637 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster 2021-11-20T08:58:44.311941+0000 mon.smithi093 ( 2021-11-20T08:58:45.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: mon.0) 638 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:boot 2021-11-20T08:58:45.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: 2021-11-20T08:58:45.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:58:45.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: 58:44.311974+0000 mon.smithi093 (mon 2021-11-20T08:58:45.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: .0) 639 : cluster [INF] Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:58:45.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster 2021-11-20T08:58:45.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: 2021-11-20T08:58: 2021-11-20T08:58:45.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: 44.312041+0000 mon.smithi093 (mon.0) 640 2021-11-20T08:58:45.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:45.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster 2021- 2021-11-20T08:58:45.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: 11-20T08:58:44 2021-11-20T08:58:45.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: .312898+0000 mon.smithi093 (mon.0) 641 : cluster 2021-11-20T08:58:45.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:45.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:58:45.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: T08:58:44.312990+0000 mon.smithi093 2021-11-20T08:58:45.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: (mon.0) 642 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:58:45.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: cluster 2021-11-20T08:58:45.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: 2021-11-20T08:58 2021-11-20T08:58:45.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: :44.317659+0000 mon.smithi093 (mon.0) 2021-11-20T08:58:45.674 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:45 smithi102 conmon[36472]: 643 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:58:45.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: cluster 2021-11-20T08:58:43.551293+0000 mgr.smithi093.qdikkm (mgr.14162) 245 : cluster [DBG] 2021-11-20T08:58:45.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: pgmap v214: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T08:58:45.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: cluster 2021-11-20T08:58:44.303766+0000 mon.smithi093 (mon.0) 637 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:58:45.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:58:45.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: 58:44.311941+0000 mon.smithi093 (mon.0) 638 2021-11-20T08:58:45.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:boot 2021-11-20T08:58:45.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: cluster 2021-11-20T08 2021-11-20T08:58:45.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: :58:44.311974+0000 mon.smithi093 (mon.0) 639 2021-11-20T08:58:45.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: : cluster [INF] Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:58:45.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:58:45.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: 08:58:44.312041+0000 mon.smithi093 (mon.0) 2021-11-20T08:58:45.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: 640 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:58:45.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T08:58:45.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: T08:58:44.312898+0000 mon.smithi093 (mon 2021-11-20T08:58:45.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: .0) 641 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:58:45.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: audit 2021 2021-11-20T08:58:45.715 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: -11-20T08:58:44.312990+0000 2021-11-20T08:58:45.716 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: mon.smithi093 (mon.0) 642 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:58:45.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:58:45.717 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: 11-20T08:58:44.317659+0000 mon.smithi093 2021-11-20T08:58:45.718 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:45 smithi093 conmon[33522]: (mon.0) 643 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:58:46.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:46.245+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:58:46.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:46.246+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:58:46.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:46.246+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:58:46.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:46.246+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:58:46.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:58:46.246+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T08:58:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:46 smithi093 conmon[33522]: cluster 2021-11-20T08:58:45.318443+0000 mon.smithi093 (mon.0) 644 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:reconnect 2021-11-20T08:58:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:46 smithi093 conmon[33522]: cluster 2021-11-20T08:58:45.318524+0000 mon.smithi093 (mon.0) 645 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:58:46.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:46 smithi093 conmon[33522]: cluster 2021-11-20T08:58:45.338304+0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:58:46.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:46 smithi102 conmon[36472]: cluster 2021-11-20T08:58:45.318443+0000 mon.smithi093 (mon.0) 644 : cluster [DBG] 2021-11-20T08:58:46.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:46 smithi102 conmon[36472]: mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:reconnect 2021-11-20T08:58:46.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:46 smithi102 conmon[36472]: cluster 2021-11-20T08:58:45.318524+0000 mon.smithi093 (mon.0) 645 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:58:46.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:46 smithi102 conmon[36472]: cluster 2021-11-20T08:58:46.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:46 smithi102 conmon[36472]: 2021-11-20T08:58:45.338304+0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:58:47.501 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:47 smithi093 conmon[33522]: cluster 2021-11-20T08:58:45.551885+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:58:47.501 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:47 smithi093 conmon[33522]: 246 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:58:47.501 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:47 smithi093 conmon[33522]: cluster 2021-11-20T08:58:46.323282+0000 mon.smithi093 (mon.0) 646 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:rejoin 2021-11-20T08:58:47.502 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:47 smithi093 conmon[33522]: cluster 2021-11-20T08:58:46.323326 2021-11-20T08:58:47.503 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:47 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 647 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:47.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:47 smithi102 conmon[36472]: cluster 2021-11-20T08:58:45.551885+0000 mgr.smithi093.qdikkm (mgr.14162) 246 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:58:47.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:47 smithi102 conmon[36472]: cluster 2021-11-20T08:58:46. 2021-11-20T08:58:47.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:47 smithi102 conmon[36472]: 323282+0000 mon.smithi093 (mon.0) 646 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1660233818,v1:172.21.15.93:6827/1660233818] up:rejoin 2021-11-20T08:58:47.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:47 smithi102 conmon[36472]: cluster 2021-11-20T08:58:46.323326+0000 mon.smithi093 (mon.0) 647 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:49.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:49 smithi102 conmon[36472]: cluster 2021-11-20T08:58:47.552424+0000 mgr.smithi093.qdikkm (mgr.14162) 247 : cluster 2021-11-20T08:58:49.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:49 smithi102 conmon[36472]: [DBG] pgmap v217: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T08:58:49.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:49 smithi102 conmon[36472]: cluster 2021-11-20T08:58:48.331750+0000 mon.smithi093 (mon.0) 648 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:standby 2021-11-20T08:58:49.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:49 smithi102 conmon[36472]: cluster 2021-11-20T08: 2021-11-20T08:58:49.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:49 smithi102 conmon[36472]: 58:48.331842+0000 mon.smithi093 (mon.0) 649 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:49.700 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:49 smithi093 conmon[33522]: cluster 2021-11-20T08:58:47.552424+0000 mgr.smithi093.qdikkm (mgr.14162) 247 : cluster 2021-11-20T08:58:49.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:49 smithi093 conmon[33522]: [DBG] pgmap v217: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T08:58:49.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:49 smithi093 conmon[33522]: cluster 2021-11-20T08:58:48.331750+0000 mon.smithi093 (mon.0) 648 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:standby 2021-11-20T08:58:49.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:49 smithi093 conmon[33522]: cluster 2021-11-20T08:58:48.331842+0000 mon.smithi093 (mon.0) 649 : 2021-11-20T08:58:49.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:49 smithi093 conmon[33522]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:58:51.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:51 smithi093 conmon[33522]: cluster 2021-11-20T08:58:49.552982+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:58:51.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:51 smithi093 conmon[33522]: 248 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:51.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:51 smithi102 conmon[36472]: cluster 2021-11-20T08:58:49.552982+0000 mgr.smithi093.qdikkm (mgr.14162) 248 : cluster 2021-11-20T08:58:51.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:51 smithi102 conmon[36472]: [DBG] pgmap v218: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:53.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:53 smithi102 conmon[36472]: cluster 2021-11-20T08:58:51.553429+0000 mgr.smithi093.qdikkm (mgr.14162) 249 : cluster [DBG] pgmap v219: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:53.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:53 smithi093 conmon[33522]: cluster 2021-11-20T08:58:51.553429+0000 mgr.smithi093.qdikkm (mgr.14162) 249 : cluster 2021-11-20T08:58:53.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:53 smithi093 conmon[33522]: [DBG] pgmap v219: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:55.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:55 smithi102 conmon[36472]: cluster 2021-11-20T08:58:53.553902+0000 mgr.smithi093.qdikkm (mgr.14162) 250 : 2021-11-20T08:58:55.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:55 smithi102 conmon[36472]: cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:55.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:55 smithi093 conmon[33522]: cluster 2021-11-20T08:58:53.553902+0000 mgr.smithi093.qdikkm (mgr.14162) 250 2021-11-20T08:58:55.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:55 smithi093 conmon[33522]: : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:58:57.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:57 smithi093 conmon[33522]: cluster 2021-11-20T08:58:55.554336+0000 mgr.smithi093.qdikkm (mgr.14162) 251 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:57.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:57 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:58:57.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:57 smithi093 conmon[33522]: 20T08:58:56.622818+0000 mon.smithi093 (mon.0) 650 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:58:57.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:57 smithi093 conmon[33522]: audit 2021-11-20T08:58:56.624097+0000 mon.smithi093 (mon.0) 651 : audit 2021-11-20T08:58:57.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:57 smithi093 conmon[33522]: [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:58:57.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:57 smithi102 conmon[36472]: cluster 2021-11-20T08:58:55.554336+0000 mgr.smithi093.qdikkm (mgr.14162) 251 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:57.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:57 smithi102 conmon[36472]: audit 2021-11-20T08:58:56.622818+0000 2021-11-20T08:58:57.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:57 smithi102 conmon[36472]: mon.smithi093 (mon.0) 650 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:58:57.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:57 smithi102 conmon[36472]: audit 2021-11-20T08:58:56.624097+0000 mon.smithi093 (mon.0) 651 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:58:59.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:58:59 smithi102 conmon[36472]: cluster 2021-11-20T08:58:57.554783+0000 mgr.smithi093.qdikkm (mgr.14162) 252 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:58:59.700 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:59 smithi093 conmon[33522]: cluster 2021-11-20T08:58:57.554783+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:58:59.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:58:59 smithi093 conmon[33522]: 252 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:59:01.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:01.262+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:01.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:01.262+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T08:59:01.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:01.262+0000 7f82a37fe700 1 -- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 msgr2=0x561580ccbb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T08:59:01.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:01.262+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T08:59:01.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:01.262+0000 7f82a37fe700 1 --2- 172.21.15.93:0/3652382463 >> [v2:172.21.15.102:6824/1423063558,v1:172.21.15.102:6825/1423063558] conn(0x561580cc97a0 0x561580ccbb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T08:59:01.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:01 smithi102 conmon[36472]: cluster 2021-11-20T08:58:59.555231+0000 mgr.smithi093.qdikkm (mgr.14162) 253 : cluster [DBG] pgmap v223: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T08:59:01.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:01 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.185194+0000 mon.smithi093 (mon.0) 652 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:59:01.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:01 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.190341+0000 mon.smithi093 (mon.0) 653 : 2021-11-20T08:59:01.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:01 smithi102 conmon[36472]: cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-20T08:59:01.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:01 smithi093 conmon[33522]: debug 2021-11-20T08:59:01.348+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:01.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:01 smithi093 conmon[33522]: cluster 2021-11-20T08:58:59.555231+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:59:01.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:01 smithi093 conmon[33522]: mgr.14162) 253 : cluster [DBG] pgmap v223: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T08:59:01.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:01 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.185194+0000 mon.smithi093 (mon.0) 652 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:59:01.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:01 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:59:01.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:01 smithi093 conmon[33522]: 59:01.190341+0000 mon.smithi093 (mon.0) 653 : cluster [DBG] 2021-11-20T08:59:01.714 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:01 smithi093 conmon[33522]: osdmap e51: 6 total, 6 up, 6 in 2021-11-20T08:59:02.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.349007+0000 mon.smithi093 (mon.0) 654 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:02.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T08:59:02.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: -20T08:59:01.352563+0000 mon.smithi093 (mon.0) 655 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:boot 2021-11-20T08:59:02.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.352644+0000 mon.smithi093 (mon.0) 656 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:59:02.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.352740+0000 mon.smithi093 (mon.0) 657 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:59:02.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.353038+0000 mon.smithi093 (mon.0) 658 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:59:02.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: audit 2021-11-20T08:59:01.353182+0000 mon.smithi093 (mon.0) 659 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:59:02.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:02 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.357539+0000 mon.smithi093 (mon.0) 660 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:59:02.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.349007+0000 mon.smithi093 (mon. 2021-11-20T08:59:02.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: 0) 654 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:02.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.352563+0000 mon.smithi093 (mon.0) 655 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:boot 2021-11-20T08:59:02.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.352644+0000 mon.smithi093 (mon.0) 656 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:59:02.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.352740+0000 mon.smithi093 (mon.0) 657 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:59:02.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.353038+0000 mon.smithi093 (mon.0) 658 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:59:02.712 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: audit 2021-11-20T08:59:01. 2021-11-20T08:59:02.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: 353182+0000 mon.smithi093 (mon.0) 659 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:59:02.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.357539 2021-11-20T08:59:02.713 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:02 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 660 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:59:03.604 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:03 smithi093 conmon[33522]: cluster 2021-11-20T08:59:01.555696+0000 2021-11-20T08:59:03.605 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:03 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 254 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:59:03.605 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:03 smithi093 conmon[33522]: cluster 2021-11-20T08:59:02.362486+0000 mon.smithi093 (mon.0) 661 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:reconnect 2021-11-20T08:59:03.606 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:03 smithi093 conmon[33522]: cluster 2021-11-20T08:59:02.362553+0000 mon.smithi093 (mon.0) 662 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:59:03.606 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:03 smithi093 conmon[33522]: cluster 2021-11-20T08:59:02.380848+0000 mds.cephfs.smithi093.zrryyo (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:59:03.607 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:03 smithi093 conmon[33522]: audit 2021-11-20T08:59:03.172255+0000 mon.smithi093 (mon.0) 663 2021-11-20T08:59:03.607 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:03 smithi093 conmon[33522]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:59:03.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: cluster 2021-11-20T08:59:01.555696+0000 2021-11-20T08:59:03.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 254 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:59:03.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: cluster 2021-11-20T08:59:02.362486+0000 2021-11-20T08:59:03.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: mon.smithi093 (mon.0) 661 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:reconnect 2021-11-20T08:59:03.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T08:59:03.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: 20T08:59:02.362553+0000 2021-11-20T08:59:03.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: mon.smithi093 (mon.0) 662 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:59:03.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T08:59:03.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: -20T08:59:02.380848+0000 mds.cephfs.smithi093.zrryyo 2021-11-20T08:59:03.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:59:03.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: audit 2021 2021-11-20T08:59:03.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: -11-20T08:59:03.172255+0000 2021-11-20T08:59:03.673 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: mon.smithi093 (mon.0) 663 : audit [DBG] 2021-11-20T08:59:03.674 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:03 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T08:59:04.460 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:04 smithi093 conmon[33522]: cluster 2021-11-20T08:59:03.362625+0000 mon.smithi093 (mon 2021-11-20T08:59:04.460 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:04 smithi093 conmon[33522]: .0) 664 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:rejoin 2021-11-20T08:59:04.460 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:04 smithi093 conmon[33522]: cluster 2021-11-20T08:59:03.362686+0000 mon.smithi093 (mon.0) 665 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:59:04.460 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:04 smithi093 conmon[33522]: audit 2021-11-20T08:59:03.599315+0000 mon.smithi093 (mon.0) 666 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:59:04.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:04 smithi102 conmon[36472]: cluster 2021-11-20T08:59:03 2021-11-20T08:59:04.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:04 smithi102 conmon[36472]: .362625+0000 mon.smithi093 (mon.0) 664 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2236115420,v1:172.21.15.93:6829/2236115420] up:rejoin 2021-11-20T08:59:04.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:04 smithi102 conmon[36472]: cluster 2021-11-20T08:59:03. 2021-11-20T08:59:04.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:04 smithi102 conmon[36472]: 362686+0000 mon.smithi093 (mon.0) 665 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:59:04.671 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:04 smithi102 conmon[36472]: audit 2021 2021-11-20T08:59:04.672 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:04 smithi102 conmon[36472]: -11-20T08:59:03.599315+0000 mon.smithi093 (mon.0) 666 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T08:59:05.655 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:05 smithi093 conmon[33522]: cluster 2021-11-20T08:59:03.556102+0000 mgr.smithi093.qdikkm (mgr.14162) 255 : cluster [DBG] 2021-11-20T08:59:05.655 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:05 smithi093 conmon[33522]: pgmap v226: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:59:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 124 2021-11-20T08:59:05.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:05 smithi102 conmon[36472]: cluster 2021-11-20T08:59:03.556102+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:59:05.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:05 smithi102 conmon[36472]: ) 255 : cluster [DBG] pgmap v226: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T08:59:06.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:06 smithi093 conmon[33522]: cluster 2021-11-20T08:59:05.377351+0000 mon.smithi093 (mon.0) 667 : cluster 2021-11-20T08:59:06.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:06 smithi093 conmon[33522]: [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:standby 2021-11-20T08:59:06.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:06 smithi093 conmon[33522]: cluster 2021-11-20T08:59:05.377413+0000 mon.smithi093 (mon.0) 668 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:59:06.658 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:06.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:06 smithi102 conmon[36472]: cluster 2021-11-20T08:59:05.377351+0000 mon.smithi093 (mon.0) 667 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:standby 2021-11-20T08:59:06.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:06 smithi102 conmon[36472]: cluster 2021-11-20T08:59:05.377413+0000 mon.smithi093 (mon.0) 668 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:59:06.790 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 -- 172.21.15.93:0/3971701147 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f4d20 msgr2=0x7fdd9c0f5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:06.791 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 --2- 172.21.15.93:0/3971701147 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f4d20 0x7fdd9c0f5140 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fdd8c0046b0 tx=0x7fdd8c00ac40).stop 2021-11-20T08:59:06.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 -- 172.21.15.93:0/3971701147 shutdown_connections 2021-11-20T08:59:06.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 --2- 172.21.15.93:0/3971701147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd9c0f5fc0 0x7fdd9c0f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:06.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 --2- 172.21.15.93:0/3971701147 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f4d20 0x7fdd9c0f5140 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:06.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 -- 172.21.15.93:0/3971701147 >> 172.21.15.93:0/3971701147 conn(0x7fdd9c0f0440 msgr2=0x7fdd9c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:06.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 -- 172.21.15.93:0/3971701147 shutdown_connections 2021-11-20T08:59:06.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.788+0000 7fdda2486700 1 -- 172.21.15.93:0/3971701147 wait complete. 2021-11-20T08:59:06.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdda2486700 1 Processor -- start 2021-11-20T08:59:06.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdda2486700 1 -- start start 2021-11-20T08:59:06.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdda2486700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd9c0f4d20 0x7fdd9c106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:06.794 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdda2486700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f5fc0 0x7fdd9c1073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:06.794 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdda2486700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c107980 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.794 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdda2486700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c107ac0 con 0x7fdd9c0f4d20 2021-11-20T08:59:06.794 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdd9b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f5fc0 0x7fdd9c1073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:06.795 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdd9b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f5fc0 0x7fdd9c1073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43592/0 (socket says 172.21.15.93:43592) 2021-11-20T08:59:06.795 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdd9b7fe700 1 -- 172.21.15.93:0/3236769374 learned_addr learned my addr 172.21.15.93:0/3236769374 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:06.795 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdd9bfff700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd9c0f4d20 0x7fdd9c106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:06.795 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdd9b7fe700 1 -- 172.21.15.93:0/3236769374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd9c0f4d20 msgr2=0x7fdd9c106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:06.796 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdd9b7fe700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd9c0f4d20 0x7fdd9c106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:06.796 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.789+0000 7fdd9b7fe700 1 -- 172.21.15.93:0/3236769374 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c005040 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.796 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.790+0000 7fdd9b7fe700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f5fc0 0x7fdd9c1073d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fdd9000e7c0 tx=0x7fdd9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:06.797 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.790+0000 7fdd98ff9700 1 -- 172.21.15.93:0/3236769374 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd900092e0 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.797 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.790+0000 7fdd98ff9700 1 -- 172.21.15.93:0/3236769374 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd9000f040 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.797 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.790+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c10c560 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.797 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.790+0000 7fdd98ff9700 1 -- 172.21.15.93:0/3236769374 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd900147c0 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.790+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c10cb20 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.791+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c0fcc20 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.791+0000 7fdd98ff9700 1 -- 172.21.15.93:0/3236769374 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdd9001b030 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.791+0000 7fdd98ff9700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8405c2a0 0x7fdd8405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:06.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.792+0000 7fdd98ff9700 1 -- 172.21.15.93:0/3236769374 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fdd900762c0 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.792+0000 7fdd9bfff700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8405c2a0 0x7fdd8405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:06.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.792+0000 7fdd9bfff700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8405c2a0 0x7fdd8405e760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fdd8c004dd0 tx=0x7fdd8c00ac40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:06.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.796+0000 7fdd98ff9700 1 -- 172.21.15.93:0/3236769374 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd90048bb0 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.980+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd9c10d1c0 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.981+0000 7fdd98ff9700 1 -- 172.21.15.93:0/3236769374 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2935 (secure 0 0 0) 0x7fdd9004c1d0 con 0x7fdd9c0f5fc0 2021-11-20T08:59:06.980 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:06.981 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:03.359337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14558":{"gid":14558,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.93:6829/2236115420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2236115420},{"type":"v1","addr":"172.21.15.93:6829","nonce":2236115420}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:06.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8405c2a0 msgr2=0x7fdd8405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:06.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8405c2a0 0x7fdd8405e760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fdd8c004dd0 tx=0x7fdd8c00ac40).stop 2021-11-20T08:59:06.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f5fc0 msgr2=0x7fdd9c1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:06.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f5fc0 0x7fdd9c1073d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fdd9000e7c0 tx=0x7fdd9000b040).stop 2021-11-20T08:59:06.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 shutdown_connections 2021-11-20T08:59:06.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8405c2a0 0x7fdd8405e760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:06.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd9c0f4d20 0x7fdd9c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:06.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 --2- 172.21.15.93:0/3236769374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd9c0f5fc0 0x7fdd9c1073d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:06.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 >> 172.21.15.93:0/3236769374 conn(0x7fdd9c0f0440 msgr2=0x7fdd9c0f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:06.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 shutdown_connections 2021-11-20T08:59:06.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:06.982+0000 7fdda2486700 1 -- 172.21.15.93:0/3236769374 wait complete. 2021-11-20T08:59:06.986 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 53 2021-11-20T08:59:06.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:03.359337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14558': {'gid': 14558, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.93:6829/2236115420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2236115420}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2236115420}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:06.994 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14558:up:rejoin 2021-11-20T08:59:07.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:07 smithi102 conmon[36472]: cluster 2021-11-20T08:59:05.556732+0000 mgr.smithi093.qdikkm (mgr.14162) 256 : cluster 2021-11-20T08:59:07.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:07 smithi102 conmon[36472]: [DBG] pgmap v227: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:07.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:07 smithi102 conmon[36472]: audit 2021-11-20T08:59:06.981589+0000 mon.smithi093 (mon.0) 669 : audit [DBG] from='client.? 172.21.15.93:0/3236769374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:07.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:07 smithi093 conmon[33522]: cluster 2021-11-20T08:59:05.556732+0000 mgr.smithi093.qdikkm (mgr.14162) 256 : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:07.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:07 smithi093 conmon[33522]: audit 2021-11-20T08:59:06.981589+0000 mon.smithi093 (mon.0) 669 : audit [DBG] from='client.? 172.21.15.93:0/3236769374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:07.995 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:08.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 -- 172.21.15.93:0/2221700780 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 msgr2=0x7f22700f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:08.124 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 --2- 172.21.15.93:0/2221700780 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f22700f5020 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f226c004660 tx=0x7f226c009b30).stop 2021-11-20T08:59:08.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 -- 172.21.15.93:0/2221700780 shutdown_connections 2021-11-20T08:59:08.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 --2- 172.21.15.93:0/2221700780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22700f5ea0 0x7f22700f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:08.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 --2- 172.21.15.93:0/2221700780 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f22700f5020 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:08.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 -- 172.21.15.93:0/2221700780 >> 172.21.15.93:0/2221700780 conn(0x7f22700f0360 msgr2=0x7f22700f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:08.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 -- 172.21.15.93:0/2221700780 shutdown_connections 2021-11-20T08:59:08.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 -- 172.21.15.93:0/2221700780 wait complete. 2021-11-20T08:59:08.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 Processor -- start 2021-11-20T08:59:08.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.122+0000 7f22762f5700 1 -- start start 2021-11-20T08:59:08.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22762f5700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f2270109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:08.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22762f5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22700f5ea0 0x7f227010bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:08.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22762f5700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22700fbe80 con 0x7f22700f4c00 2021-11-20T08:59:08.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22762f5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22700fbfc0 con 0x7f22700f5ea0 2021-11-20T08:59:08.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22752f3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f2270109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:08.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22752f3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f2270109a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43600/0 (socket says 172.21.15.93:43600) 2021-11-20T08:59:08.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22752f3700 1 -- 172.21.15.93:0/2441266742 learned_addr learned my addr 172.21.15.93:0/2441266742 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:08.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f2274af2700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22700f5ea0 0x7f227010bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:08.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22752f3700 1 -- 172.21.15.93:0/2441266742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22700f5ea0 msgr2=0x7f227010bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:08.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22752f3700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22700f5ea0 0x7f227010bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:08.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22752f3700 1 -- 172.21.15.93:0/2441266742 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c005040 con 0x7f22700f4c00 2021-11-20T08:59:08.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22752f3700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f2270109a00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f226c000c00 tx=0x7f226c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:08.131 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f2265ffb700 1 -- 172.21.15.93:0/2441266742 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226c019070 con 0x7f22700f4c00 2021-11-20T08:59:08.131 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f2265ffb700 1 -- 172.21.15.93:0/2441266742 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f226c007e70 con 0x7f22700f4c00 2021-11-20T08:59:08.131 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227010c450 con 0x7f22700f4c00 2021-11-20T08:59:08.131 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f2265ffb700 1 -- 172.21.15.93:0/2441266742 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226c014d40 con 0x7f22700f4c00 2021-11-20T08:59:08.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.123+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227010c980 con 0x7f22700f4c00 2021-11-20T08:59:08.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.124+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22700f3ff0 con 0x7f22700f4c00 2021-11-20T08:59:08.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.125+0000 7f2265ffb700 1 -- 172.21.15.93:0/2441266742 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f226c025070 con 0x7f22700f4c00 2021-11-20T08:59:08.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.125+0000 7f2265ffb700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f225c05c300 0x7f225c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:08.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.125+0000 7f2265ffb700 1 -- 172.21.15.93:0/2441266742 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f226c0770f0 con 0x7f22700f4c00 2021-11-20T08:59:08.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.125+0000 7f2274af2700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f225c05c300 0x7f225c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:08.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.125+0000 7f2274af2700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f225c05c300 0x7f225c05e7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f2260009fd0 tx=0x7f226000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:08.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.129+0000 7f2265ffb700 1 -- 172.21.15.93:0/2441266742 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f226c049b00 con 0x7f22700f4c00 2021-11-20T08:59:08.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.320+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2270045ad0 con 0x7f22700f4c00 2021-11-20T08:59:08.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.320+0000 7f2265ffb700 1 -- 172.21.15.93:0/2441266742 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2935 (secure 0 0 0) 0x7f226c04d120 con 0x7f22700f4c00 2021-11-20T08:59:08.321 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:08.321 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:03.359337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14558":{"gid":14558,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.93:6829/2236115420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2236115420},{"type":"v1","addr":"172.21.15.93:6829","nonce":2236115420}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:08.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f225c05c300 msgr2=0x7f225c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:08.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f225c05c300 0x7f225c05e7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f2260009fd0 tx=0x7f226000b040).stop 2021-11-20T08:59:08.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 msgr2=0x7f2270109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f2270109a00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f226c000c00 tx=0x7f226c00b040).stop 2021-11-20T08:59:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 shutdown_connections 2021-11-20T08:59:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f225c05c300 0x7f225c05e7c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22700f4c00 0x7f2270109a00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 --2- 172.21.15.93:0/2441266742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22700f5ea0 0x7f227010bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 >> 172.21.15.93:0/2441266742 conn(0x7f22700f0360 msgr2=0x7f22700f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 shutdown_connections 2021-11-20T08:59:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:08.322+0000 7f22762f5700 1 -- 172.21.15.93:0/2441266742 wait complete. 2021-11-20T08:59:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 53 2021-11-20T08:59:08.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:03.359337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14558': {'gid': 14558, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.93:6829/2236115420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2236115420}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2236115420}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:08.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14558:up:rejoin 2021-11-20T08:59:08.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:08 smithi102 conmon[36472]: audit 2021-11-20T08:59 2021-11-20T08:59:08.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:08 smithi102 conmon[36472]: :08.321137+0000 mon.smithi093 (mon.0) 670 : audit [DBG] from='client.? 172.21.15.93:0/2441266742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:08.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:08 smithi093 conmon[33522]: audit 2021-11-20T08:59:08.321137+0000 mon.smithi093 (mon 2021-11-20T08:59:08.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:08 smithi093 conmon[33522]: .0) 670 : audit [DBG] from='client.? 172.21.15.93:0/2441266742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:09.335 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:09.465 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 -- 172.21.15.93:0/4271163116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f7550 msgr2=0x7efcec0f5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:09.466 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 --2- 172.21.15.93:0/4271163116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f7550 0x7efcec0f5970 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efcdc004660 tx=0x7efcdc009b30).stop 2021-11-20T08:59:09.466 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 -- 172.21.15.93:0/4271163116 shutdown_connections 2021-11-20T08:59:09.466 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 --2- 172.21.15.93:0/4271163116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f7550 0x7efcec0f5970 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:09.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 --2- 172.21.15.93:0/4271163116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcec0f6bb0 0x7efcec0f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:09.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 -- 172.21.15.93:0/4271163116 >> 172.21.15.93:0/4271163116 conn(0x7efcec0f0440 msgr2=0x7efcec0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:09.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 -- 172.21.15.93:0/4271163116 shutdown_connections 2021-11-20T08:59:09.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.463+0000 7efcf46de700 1 -- 172.21.15.93:0/4271163116 wait complete. 2021-11-20T08:59:09.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf46de700 1 Processor -- start 2021-11-20T08:59:09.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf46de700 1 -- start start 2021-11-20T08:59:09.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf46de700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f6bb0 0x7efcec0f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:09.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf46de700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcec0f7550 0x7efcec10bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:09.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf46de700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec10c140 con 0x7efcec0f6bb0 2021-11-20T08:59:09.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf46de700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec10c280 con 0x7efcec0f7550 2021-11-20T08:59:09.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf247a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f6bb0 0x7efcec0f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:09.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf247a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f6bb0 0x7efcec0f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43612/0 (socket says 172.21.15.93:43612) 2021-11-20T08:59:09.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf247a700 1 -- 172.21.15.93:0/2412665295 learned_addr learned my addr 172.21.15.93:0/2412665295 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:09.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf1c79700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcec0f7550 0x7efcec10bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:09.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf247a700 1 -- 172.21.15.93:0/2412665295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcec0f7550 msgr2=0x7efcec10bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:09.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf247a700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcec0f7550 0x7efcec10bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:09.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.464+0000 7efcf247a700 1 -- 172.21.15.93:0/2412665295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc005040 con 0x7efcec0f6bb0 2021-11-20T08:59:09.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.465+0000 7efcf247a700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f6bb0 0x7efcec0f5310 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7efce800d820 tx=0x7efce8004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:09.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.465+0000 7efce2ffd700 1 -- 172.21.15.93:0/2412665295 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce800b510 con 0x7efcec0f6bb0 2021-11-20T08:59:09.472 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.465+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec10c4e0 con 0x7efcec0f6bb0 2021-11-20T08:59:09.472 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.465+0000 7efce2ffd700 1 -- 172.21.15.93:0/2412665295 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efce800f040 con 0x7efcec0f6bb0 2021-11-20T08:59:09.472 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.465+0000 7efce2ffd700 1 -- 172.21.15.93:0/2412665295 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce8013a30 con 0x7efcec0f6bb0 2021-11-20T08:59:09.472 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.465+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec10caa0 con 0x7efcec0f6bb0 2021-11-20T08:59:09.473 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.466+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec027a00 con 0x7efcec0f6bb0 2021-11-20T08:59:09.473 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.466+0000 7efce2ffd700 1 -- 172.21.15.93:0/2412665295 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efce801a030 con 0x7efcec0f6bb0 2021-11-20T08:59:09.474 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.466+0000 7efce2ffd700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7efcd805c2a0 0x7efcd805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:09.474 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.466+0000 7efce2ffd700 1 -- 172.21.15.93:0/2412665295 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6070+0+0 (secure 0 0 0) 0x7efce8076040 con 0x7efcec0f6bb0 2021-11-20T08:59:09.474 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.466+0000 7efcf1c79700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7efcd805c2a0 0x7efcd805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:09.475 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.467+0000 7efcf1c79700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7efcd805c2a0 0x7efcd805e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7efcdc004e30 tx=0x7efcdc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:09.476 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.470+0000 7efce2ffd700 1 -- 172.21.15.93:0/2412665295 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efce8048a50 con 0x7efcec0f6bb0 2021-11-20T08:59:09.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.654+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcec0fdd40 con 0x7efcec0f6bb0 2021-11-20T08:59:09.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.655+0000 7efce2ffd700 1 -- 172.21.15.93:0/2412665295 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2935 (secure 0 0 0) 0x7efce8018020 con 0x7efcec0f6bb0 2021-11-20T08:59:09.655 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:09.655 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:03.359337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14558":{"gid":14558,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.93:6829/2236115420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2236115420},{"type":"v1","addr":"172.21.15.93:6829","nonce":2236115420}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:09.657 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7efcd805c2a0 msgr2=0x7efcd805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:09.657 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7efcd805c2a0 0x7efcd805e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7efcdc004e30 tx=0x7efcdc009b30).stop 2021-11-20T08:59:09.657 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f6bb0 msgr2=0x7efcec0f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:09.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f6bb0 0x7efcec0f5310 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7efce800d820 tx=0x7efce8004e80).stop 2021-11-20T08:59:09.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 shutdown_connections 2021-11-20T08:59:09.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7efcd805c2a0 0x7efcd805e760 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:09.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efcec0f6bb0 0x7efcec0f5310 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:09.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 --2- 172.21.15.93:0/2412665295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcec0f7550 0x7efcec10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:09.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 >> 172.21.15.93:0/2412665295 conn(0x7efcec0f0440 msgr2=0x7efcec0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:09.663 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 shutdown_connections 2021-11-20T08:59:09.663 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:09.657+0000 7efcf46de700 1 -- 172.21.15.93:0/2412665295 wait complete. 2021-11-20T08:59:09.664 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 53 2021-11-20T08:59:09.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:09 smithi093 conmon[33522]: cluster 2021-11-20T08:59:07.557107+0000 mgr.smithi093.qdikkm (mgr.14162) 257 : cluster 2021-11-20T08:59:09.667 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:09 smithi093 conmon[33522]: [DBG] pgmap v228: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:09.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:09 smithi102 conmon[36472]: cluster 2021-11-20T08:59:07.557107+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:59:09.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:09 smithi102 conmon[36472]: .14162) 257 : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:09.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:03.359337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14558': {'gid': 14558, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.93:6829/2236115420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2236115420}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2236115420}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:09.672 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14558:up:rejoin 2021-11-20T08:59:10.667 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:10 smithi102 conmon[36472]: audit 2021-11-20T08:59:09.655996+0000 mon.smithi093 (mon.0 2021-11-20T08:59:10.668 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:10 smithi102 conmon[36472]: ) 671 : audit [DBG] from='client.? 172.21.15.93:0/2412665295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:10.673 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:10.700 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:10 smithi093 conmon[33522]: audit 2021-11-20T08:59:09.655996+0000 mon.smithi093 (mon.0) 671 2021-11-20T08:59:10.701 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:10 smithi093 conmon[33522]: : audit [DBG] from='client.? 172.21.15.93:0/2412665295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:10.801 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 -- 172.21.15.93:0/3367923029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f6700 msgr2=0x7fa1980f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:10.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 --2- 172.21.15.93:0/3367923029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f6700 0x7fa1980f6b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fa18c004660 tx=0x7fa18c009b30).stop 2021-11-20T08:59:10.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 -- 172.21.15.93:0/3367923029 shutdown_connections 2021-11-20T08:59:10.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 --2- 172.21.15.93:0/3367923029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f6700 0x7fa1980f6b60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:10.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 --2- 172.21.15.93:0/3367923029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1980f5460 0x7fa1980f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:10.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 -- 172.21.15.93:0/3367923029 >> 172.21.15.93:0/3367923029 conn(0x7fa1980f0ba0 msgr2=0x7fa1980f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:10.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 -- 172.21.15.93:0/3367923029 shutdown_connections 2021-11-20T08:59:10.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 -- 172.21.15.93:0/3367923029 wait complete. 2021-11-20T08:59:10.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 Processor -- start 2021-11-20T08:59:10.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.800+0000 7fa19d59e700 1 -- start start 2021-11-20T08:59:10.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa19d59e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f5460 0x7fa198105c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:10.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa19d59e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1980f6700 0x7fa198106100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:10.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa19d59e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981066b0 con 0x7fa1980f5460 2021-11-20T08:59:10.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa19d59e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981067f0 con 0x7fa1980f6700 2021-11-20T08:59:10.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa197fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f5460 0x7fa198105c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:10.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa197fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f5460 0x7fa198105c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43622/0 (socket says 172.21.15.93:43622) 2021-11-20T08:59:10.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa197fff700 1 -- 172.21.15.93:0/2408483237 learned_addr learned my addr 172.21.15.93:0/2408483237 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:10.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa1977fe700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1980f6700 0x7fa198106100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:10.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa197fff700 1 -- 172.21.15.93:0/2408483237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1980f6700 msgr2=0x7fa198106100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:10.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa197fff700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1980f6700 0x7fa198106100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:10.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa197fff700 1 -- 172.21.15.93:0/2408483237 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c005040 con 0x7fa1980f5460 2021-11-20T08:59:10.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa197fff700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f5460 0x7fa198105c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fa18800c530 tx=0x7fa188007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:10.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa194ff9700 1 -- 172.21.15.93:0/2408483237 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa18800f650 con 0x7fa1980f5460 2021-11-20T08:59:10.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa198107190 con 0x7fa1980f5460 2021-11-20T08:59:10.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa194ff9700 1 -- 172.21.15.93:0/2408483237 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa18800fc60 con 0x7fa1980f5460 2021-11-20T08:59:10.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa194ff9700 1 -- 172.21.15.93:0/2408483237 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa188012d20 con 0x7fa1980f5460 2021-11-20T08:59:10.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.801+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa198107750 con 0x7fa1980f5460 2021-11-20T08:59:10.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.803+0000 7fa194ff9700 1 -- 172.21.15.93:0/2408483237 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa188019030 con 0x7fa1980f5460 2021-11-20T08:59:10.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.803+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa198045ad0 con 0x7fa1980f5460 2021-11-20T08:59:10.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.803+0000 7fa194ff9700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa18005c300 0x7fa18005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:10.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.803+0000 7fa194ff9700 1 -- 172.21.15.93:0/2408483237 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fa188075e20 con 0x7fa1980f5460 2021-11-20T08:59:10.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.803+0000 7fa1977fe700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa18005c300 0x7fa18005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:10.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.803+0000 7fa1977fe700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa18005c300 0x7fa18005e7c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa18c013580 tx=0x7fa18c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:10.810 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.807+0000 7fa194ff9700 1 -- 172.21.15.93:0/2408483237 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa188048880 con 0x7fa1980f5460 2021-11-20T08:59:10.990 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.991+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa198107df0 con 0x7fa1980f5460 2021-11-20T08:59:10.991 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.992+0000 7fa194ff9700 1 -- 172.21.15.93:0/2408483237 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2935 (secure 0 0 0) 0x7fa188017070 con 0x7fa1980f5460 2021-11-20T08:59:10.992 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:10.992 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:03.359337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14558":{"gid":14558,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.93:6829/2236115420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2236115420},{"type":"v1","addr":"172.21.15.93:6829","nonce":2236115420}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:10.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa18005c300 msgr2=0x7fa18005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:10.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa18005c300 0x7fa18005e7c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa18c013580 tx=0x7fa18c009b30).stop 2021-11-20T08:59:10.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f5460 msgr2=0x7fa198105c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:10.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f5460 0x7fa198105c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fa18800c530 tx=0x7fa188007e10).stop 2021-11-20T08:59:10.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 shutdown_connections 2021-11-20T08:59:10.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa18005c300 0x7fa18005e7c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:10.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1980f5460 0x7fa198105c10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:10.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 --2- 172.21.15.93:0/2408483237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1980f6700 0x7fa198106100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:10.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 >> 172.21.15.93:0/2408483237 conn(0x7fa1980f0ba0 msgr2=0x7fa1980fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:10.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 shutdown_connections 2021-11-20T08:59:10.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:10.994+0000 7fa19d59e700 1 -- 172.21.15.93:0/2408483237 wait complete. 2021-11-20T08:59:10.996 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 53 2021-11-20T08:59:11.004 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:03.359337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14558': {'gid': 14558, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.93:6829/2236115420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2236115420}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2236115420}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:11.005 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14558:up:rejoin 2021-11-20T08:59:11.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:11 smithi102 conmon[36472]: cluster 2021-11-20T08:59:09.557468+0000 mgr.smithi093.qdikkm (mgr.14162) 258 2021-11-20T08:59:11.669 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:11 smithi102 conmon[36472]: : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:11.670 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:11 smithi102 conmon[36472]: audit 2021-11-20T08:59:10.992972+0000 mon.smithi093 (mon.0) 672 : audit [DBG] from='client.? 172.21.15.93:0/2408483237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:11.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:11 smithi093 conmon[33522]: cluster 2021-11-20T08:59:09.557468+0000 mgr.smithi093.qdikkm (mgr.14162) 258 : cluster 2021-11-20T08:59:11.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:11 smithi093 conmon[33522]: [DBG] pgmap v229: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:11.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:11 smithi093 conmon[33522]: audit 2021-11-20T08:59:10.992972+0000 mon.smithi093 (mon.0) 672 : audit [DBG] from='client.? 172.21.15.93:0/2408483237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:12.006 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:12.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- 172.21.15.93:0/4105253874 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 msgr2=0x7f33100f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:12.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 --2- 172.21.15.93:0/4105253874 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f33100f59c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f330c004660 tx=0x7f330c009b30).stop 2021-11-20T08:59:12.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- 172.21.15.93:0/4105253874 shutdown_connections 2021-11-20T08:59:12.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 --2- 172.21.15.93:0/4105253874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f6840 0x7f33100f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:12.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 --2- 172.21.15.93:0/4105253874 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f33100f59c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:12.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- 172.21.15.93:0/4105253874 >> 172.21.15.93:0/4105253874 conn(0x7f33100f0ce0 msgr2=0x7f33100f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:12.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- 172.21.15.93:0/4105253874 shutdown_connections 2021-11-20T08:59:12.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- 172.21.15.93:0/4105253874 wait complete. 2021-11-20T08:59:12.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 Processor -- start 2021-11-20T08:59:12.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- start start 2021-11-20T08:59:12.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f3310106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:12.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f6840 0x7f33101073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:12.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3310107970 con 0x7f33100f55a0 2021-11-20T08:59:12.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.132+0000 7f3318288700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3310107ab0 con 0x7f33100f6840 2021-11-20T08:59:12.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3316024700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f3310106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:12.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3316024700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f3310106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43632/0 (socket says 172.21.15.93:43632) 2021-11-20T08:59:12.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3316024700 1 -- 172.21.15.93:0/3961076109 learned_addr learned my addr 172.21.15.93:0/3961076109 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:12.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3315823700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f6840 0x7f33101073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:12.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3316024700 1 -- 172.21.15.93:0/3961076109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f6840 msgr2=0x7f33101073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:12.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3316024700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f6840 0x7f33101073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:12.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3316024700 1 -- 172.21.15.93:0/3961076109 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c005040 con 0x7f33100f55a0 2021-11-20T08:59:12.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3316024700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f3310106ed0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f330c009f50 tx=0x7f330c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:12.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3306ffd700 1 -- 172.21.15.93:0/3961076109 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f330c018070 con 0x7f33100f55a0 2021-11-20T08:59:12.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3306ffd700 1 -- 172.21.15.93:0/3961076109 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f330c013410 con 0x7f33100f55a0 2021-11-20T08:59:12.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331010c4f0 con 0x7f33100f55a0 2021-11-20T08:59:12.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3306ffd700 1 -- 172.21.15.93:0/3961076109 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f330c01c620 con 0x7f33100f55a0 2021-11-20T08:59:12.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.133+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331010cab0 con 0x7f33100f55a0 2021-11-20T08:59:12.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.134+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3310027a00 con 0x7f33100f55a0 2021-11-20T08:59:12.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.134+0000 7f3306ffd700 1 -- 172.21.15.93:0/3961076109 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f330c01d420 con 0x7f33100f55a0 2021-11-20T08:59:12.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.135+0000 7f3306ffd700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c2a0 0x7f32fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:12.142 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.135+0000 7f3306ffd700 1 -- 172.21.15.93:0/3961076109 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f330c077fe0 con 0x7f33100f55a0 2021-11-20T08:59:12.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.135+0000 7f3315823700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c2a0 0x7f32fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:12.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.135+0000 7f3315823700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c2a0 0x7f32fc05e760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f33000041d0 tx=0x7f3300009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:12.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.139+0000 7f3306ffd700 1 -- 172.21.15.93:0/3961076109 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f330c04a9f0 con 0x7f33100f55a0 2021-11-20T08:59:12.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.331+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33100fee30 con 0x7f33100f55a0 2021-11-20T08:59:12.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.332+0000 7f3306ffd700 1 -- 172.21.15.93:0/3961076109 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2935 (secure 0 0 0) 0x7f330c04e010 con 0x7f33100f55a0 2021-11-20T08:59:12.331 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:12.332 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:03.359337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14558":{"gid":14558,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.93:6829/2236115420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2236115420},{"type":"v1","addr":"172.21.15.93:6829","nonce":2236115420}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:12.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c2a0 msgr2=0x7f32fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:12.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c2a0 0x7f32fc05e760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f33000041d0 tx=0x7f3300009040).stop 2021-11-20T08:59:12.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 msgr2=0x7f3310106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:12.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f3310106ed0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f330c009f50 tx=0x7f330c009b30).stop 2021-11-20T08:59:12.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 shutdown_connections 2021-11-20T08:59:12.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c2a0 0x7f32fc05e760 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:12.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f55a0 0x7f3310106ed0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:12.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 --2- 172.21.15.93:0/3961076109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f6840 0x7f33101073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:12.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 >> 172.21.15.93:0/3961076109 conn(0x7f33100f0ce0 msgr2=0x7f33100fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:12.336 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 shutdown_connections 2021-11-20T08:59:12.337 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:12.334+0000 7f3318288700 1 -- 172.21.15.93:0/3961076109 wait complete. 2021-11-20T08:59:12.337 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 53 2021-11-20T08:59:12.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:03.359337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14558': {'gid': 14558, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.93:6829/2236115420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2236115420}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2236115420}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:12.345 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14558:up:rejoin 2021-11-20T08:59:12.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:12 smithi093 conmon[33522]: cluster 2021-11-20T08:59:11.557982+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:59:12.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:12 smithi093 conmon[33522]: ) 259 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:12.711 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:12 smithi093 conmon[33522]: audit 2021-11-20T08:59:12.332929+0000 mon.smithi093 (mon.0) 673 : audit [DBG] from='client.? 172.21.15.93:0/3961076109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:12.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:12 smithi102 conmon[36472]: cluster 2021-11-20T08:59:11.557982+0000 mgr.smithi093.qdikkm (mgr.14162) 259 : cluster [DBG] 2021-11-20T08:59:12.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:12 smithi102 conmon[36472]: pgmap v230: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:12.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:12 smithi102 conmon[36472]: audit 2021-11-20T08:59:12.332929+0000 mon.smithi093 (mon.0) 673 : audit [DBG] from='client.? 172.21.15.93:0/3961076109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:13.346 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:13.480 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 -- 172.21.15.93:0/2072373565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 msgr2=0x7fce280f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:13.481 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/2072373565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce280f7aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fce1c004660 tx=0x7fce1c009b30).stop 2021-11-20T08:59:13.481 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 -- 172.21.15.93:0/2072373565 shutdown_connections 2021-11-20T08:59:13.482 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/2072373565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce280f7aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:13.482 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/2072373565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce280f4980 0x7fce280f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:13.482 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 -- 172.21.15.93:0/2072373565 >> 172.21.15.93:0/2072373565 conn(0x7fce280f04a0 msgr2=0x7fce280f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:13.483 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 -- 172.21.15.93:0/2072373565 shutdown_connections 2021-11-20T08:59:13.483 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 -- 172.21.15.93:0/2072373565 wait complete. 2021-11-20T08:59:13.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 Processor -- start 2021-11-20T08:59:13.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 -- start start 2021-11-20T08:59:13.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce280f4980 0x7fce28106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:13.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.475+0000 7fce2dfcc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce28107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:13.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce2dfcc700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce28107900 con 0x7fce280f5320 2021-11-20T08:59:13.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce2dfcc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce28107a40 con 0x7fce280f4980 2021-11-20T08:59:13.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce26ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce28107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:13.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce26ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce28107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43646/0 (socket says 172.21.15.93:43646) 2021-11-20T08:59:13.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce26ffd700 1 -- 172.21.15.93:0/3829792557 learned_addr learned my addr 172.21.15.93:0/3829792557 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:13.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce277fe700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce280f4980 0x7fce28106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:13.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce26ffd700 1 -- 172.21.15.93:0/3829792557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce280f4980 msgr2=0x7fce28106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:13.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce26ffd700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce280f4980 0x7fce28106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:13.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce26ffd700 1 -- 172.21.15.93:0/3829792557 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c005040 con 0x7fce280f5320 2021-11-20T08:59:13.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce26ffd700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce28107350 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fce1c004e30 tx=0x7fce1c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:13.492 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce0ffff700 1 -- 172.21.15.93:0/3829792557 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1c018070 con 0x7fce280f5320 2021-11-20T08:59:13.493 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce0ffff700 1 -- 172.21.15.93:0/3829792557 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce1c013410 con 0x7fce280f5320 2021-11-20T08:59:13.493 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2810c480 con 0x7fce280f5320 2021-11-20T08:59:13.493 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce0ffff700 1 -- 172.21.15.93:0/3829792557 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1c01c3c0 con 0x7fce280f5320 2021-11-20T08:59:13.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.476+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2810ca40 con 0x7fce280f5320 2021-11-20T08:59:13.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.477+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce28027a00 con 0x7fce280f5320 2021-11-20T08:59:13.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.478+0000 7fce0ffff700 1 -- 172.21.15.93:0/3829792557 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fce1c01d420 con 0x7fce280f5320 2021-11-20T08:59:13.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.478+0000 7fce0ffff700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce1005c2a0 0x7fce1005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:13.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.478+0000 7fce0ffff700 1 -- 172.21.15.93:0/3829792557 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fce1c077d70 con 0x7fce280f5320 2021-11-20T08:59:13.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.478+0000 7fce277fe700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce1005c2a0 0x7fce1005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:13.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.478+0000 7fce277fe700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce1005c2a0 0x7fce1005e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fce18009880 tx=0x7fce1800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:13.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.482+0000 7fce0ffff700 1 -- 172.21.15.93:0/3829792557 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce1c04a780 con 0x7fce280f5320 2021-11-20T08:59:13.665 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.666+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce280fee70 con 0x7fce280f5320 2021-11-20T08:59:13.666 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.667+0000 7fce0ffff700 1 -- 172.21.15.93:0/3829792557 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2935 (secure 0 0 0) 0x7fce1c04dda0 con 0x7fce280f5320 2021-11-20T08:59:13.666 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:13.666 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:03.359337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14558},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14558":{"gid":14558,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.93:6829/2236115420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2236115420},{"type":"v1","addr":"172.21.15.93:6829","nonce":2236115420}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:13.668 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce1005c2a0 msgr2=0x7fce1005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:13.668 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce1005c2a0 0x7fce1005e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fce18009880 tx=0x7fce1800b040).stop 2021-11-20T08:59:13.668 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 msgr2=0x7fce28107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:13.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce28107350 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fce1c004e30 tx=0x7fce1c009b30).stop 2021-11-20T08:59:13.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 shutdown_connections 2021-11-20T08:59:13.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce1005c2a0 0x7fce1005e760 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:13.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce280f4980 0x7fce28106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:13.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 --2- 172.21.15.93:0/3829792557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce280f5320 0x7fce28107350 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:13.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 >> 172.21.15.93:0/3829792557 conn(0x7fce280f04a0 msgr2=0x7fce280fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:13.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.668+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 shutdown_connections 2021-11-20T08:59:13.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:13.669+0000 7fce2dfcc700 1 -- 172.21.15.93:0/3829792557 wait complete. 2021-11-20T08:59:13.671 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 53 2021-11-20T08:59:13.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:03.359337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14558}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14558': {'gid': 14558, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.93:6829/2236115420', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2236115420}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2236115420}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:13.681 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14558:up:rejoin 2021-11-20T08:59:14.682 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:14.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 -- 172.21.15.93:0/2194093744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f57c0 msgr2=0x7fa1140f5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:14.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 --2- 172.21.15.93:0/2194093744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f57c0 0x7fa1140f5be0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fa108004660 tx=0x7fa108009b30).stop 2021-11-20T08:59:14.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 -- 172.21.15.93:0/2194093744 shutdown_connections 2021-11-20T08:59:14.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 --2- 172.21.15.93:0/2194093744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1140f6a60 0x7fa1140f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:14.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 --2- 172.21.15.93:0/2194093744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f57c0 0x7fa1140f5be0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:14.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 -- 172.21.15.93:0/2194093744 >> 172.21.15.93:0/2194093744 conn(0x7fa1140f0ec0 msgr2=0x7fa1140f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:14.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 -- 172.21.15.93:0/2194093744 shutdown_connections 2021-11-20T08:59:14.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 -- 172.21.15.93:0/2194093744 wait complete. 2021-11-20T08:59:14.810 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 Processor -- start 2021-11-20T08:59:14.810 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.802+0000 7fa11afc4700 1 -- start start 2021-11-20T08:59:14.810 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa11afc4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1140f57c0 0x7fa114105fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:14.811 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa11afc4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f6a60 0x7fa1141064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:14.811 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa11afc4700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa114106a70 con 0x7fa1140f6a60 2021-11-20T08:59:14.811 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa11afc4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa114106bb0 con 0x7fa1140f57c0 2021-11-20T08:59:14.812 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa113fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f6a60 0x7fa1141064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:14.812 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa113fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f6a60 0x7fa1141064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43676/0 (socket says 172.21.15.93:43676) 2021-11-20T08:59:14.812 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa113fff700 1 -- 172.21.15.93:0/58297578 learned_addr learned my addr 172.21.15.93:0/58297578 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:14.813 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa113fff700 1 -- 172.21.15.93:0/58297578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1140f57c0 msgr2=0x7fa114105fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:14.813 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa118d60700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1140f57c0 0x7fa114105fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:14.813 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa113fff700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1140f57c0 0x7fa114105fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:14.814 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa113fff700 1 -- 172.21.15.93:0/58297578 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108005040 con 0x7fa1140f6a60 2021-11-20T08:59:14.814 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa113fff700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f6a60 0x7fa1141064c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fa10000c790 tx=0x7fa100007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:14.814 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa1117fa700 1 -- 172.21.15.93:0/58297578 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa10000e680 con 0x7fa1140f6a60 2021-11-20T08:59:14.815 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa1117fa700 1 -- 172.21.15.93:0/58297578 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa10000ec90 con 0x7fa1140f6a60 2021-11-20T08:59:14.815 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11410b650 con 0x7fa1140f6a60 2021-11-20T08:59:14.815 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa1117fa700 1 -- 172.21.15.93:0/58297578 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa100013d60 con 0x7fa1140f6a60 2021-11-20T08:59:14.816 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.803+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11410bc10 con 0x7fa1140f6a60 2021-11-20T08:59:14.816 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.804+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11405ea10 con 0x7fa1140f6a60 2021-11-20T08:59:14.816 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.805+0000 7fa1117fa700 1 -- 172.21.15.93:0/58297578 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa10001a060 con 0x7fa1140f6a60 2021-11-20T08:59:14.816 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.805+0000 7fa1117fa700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa10405c2a0 0x7fa10405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:14.817 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.805+0000 7fa1117fa700 1 -- 172.21.15.93:0/58297578 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa100076410 con 0x7fa1140f6a60 2021-11-20T08:59:14.817 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.805+0000 7fa118d60700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa10405c2a0 0x7fa10405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:14.818 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.805+0000 7fa118d60700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa10405c2a0 0x7fa10405e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa108009d60 tx=0x7fa108009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:14.819 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.809+0000 7fa1117fa700 1 -- 172.21.15.93:0/58297578 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa100048dd0 con 0x7fa1140f6a60 2021-11-20T08:59:14.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:14 smithi102 conmon[36472]: cluster 2021-11-20T08:59:13.558328+0000 mgr.smithi093.qdikkm (mgr.14162) 260 : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T08:59:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:14 smithi102 conmon[36472]: 2021-11-20T08:59:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:14 smithi102 conmon[36472]: audit 2021-11-20T08:59:13.667755+0000 mon.smithi093 (mon.0) 674 : audit [DBG] from='client.? 172.21.15.93:0/3829792557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:14.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:14 smithi102 conmon[36472]: cluster 2021-11-20T08:59:14.167965+0000 mon.smithi093 (mon.0) 675 : cluster [INF] 2021-11-20T08:59:14.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:14 smithi102 conmon[36472]: daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:59:14.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:14 smithi102 conmon[36472]: cluster 2021-11-20T08:59:14.173694+0000 mon.smithi093 (mon.0) 676 : cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-20T08:59:14.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: debug 2021-11-20T08:59:14.586+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:14.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: cluster 2021-11-20T08:59:13.558328+0000 mgr.smithi093.qdikkm (mgr.14162) 260 : cluster [DBG] 2021-11-20T08:59:14.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: pgmap v231: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-20T08:59:14.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: audit 2021-11-20T08:59:13.667755+0000 mon.smithi093 (mon.0) 674 : 2021-11-20T08:59:14.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: audit [DBG] from='client.? 172.21.15.93:0/3829792557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:14.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:59:14.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: 20T08:59:14.167965+0000 mon.smithi093 (mon.0) 675 2021-11-20T08:59:14.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: : cluster [INF] daemon mds.cephfs.smithi093.zrryyo restarted 2021-11-20T08:59:14.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:59:14.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: 08:59:14.173694+0000 mon.smithi093 (mon 2021-11-20T08:59:14.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:14 smithi093 conmon[33522]: .0) 676 : cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-20T08:59:14.992 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.993+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa114027a00 con 0x7fa1140f6a60 2021-11-20T08:59:14.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.994+0000 7fa1117fa700 1 -- 172.21.15.93:0/58297578 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2936 (secure 0 0 0) 0x7fa10004c3f0 con 0x7fa1140f6a60 2021-11-20T08:59:14.993 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:14.993 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:14.595421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:replay","state_seq":3,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:14.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.995+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa10405c2a0 msgr2=0x7fa10405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:14.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.995+0000 7fa11afc4700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa10405c2a0 0x7fa10405e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa108009d60 tx=0x7fa108009b30).stop 2021-11-20T08:59:14.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.995+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f6a60 msgr2=0x7fa1141064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:14.995 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.995+0000 7fa11afc4700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f6a60 0x7fa1141064c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fa10000c790 tx=0x7fa100007d70).stop 2021-11-20T08:59:14.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.996+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 shutdown_connections 2021-11-20T08:59:14.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.996+0000 7fa11afc4700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa10405c2a0 0x7fa10405e760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:14.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.996+0000 7fa11afc4700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1140f57c0 0x7fa114105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:14.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.996+0000 7fa11afc4700 1 --2- 172.21.15.93:0/58297578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1140f6a60 0x7fa1141064c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:14.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.996+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 >> 172.21.15.93:0/58297578 conn(0x7fa1140f0ec0 msgr2=0x7fa1140fa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:14.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.996+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 shutdown_connections 2021-11-20T08:59:14.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:14.996+0000 7fa11afc4700 1 -- 172.21.15.93:0/58297578 wait complete. 2021-11-20T08:59:14.997 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 55 2021-11-20T08:59:15.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:14.595421+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:15.007 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:replay 2021-11-20T08:59:15.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: cluster 2021-11-20T08:59:14.587567+0000 mon.smithi093 (mon.0) 677 : cluster 2021-11-20T08:59:15.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:15.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: cluster 2021-11-20T08:59:14.595341+0000 mon.smithi093 (mon.0) 678 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:boot 2021-11-20T08:59:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T08:59:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: 20T08:59:14.595381+0000 mon.smithi093 (mon.0) 679 : cluster [INF] Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:59:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: cluster 2021-11-20T08:59:14.595480+0000 2021-11-20T08:59:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: mon.smithi093 (mon.0) 680 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:59:15.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: cluster 2021- 2021-11-20T08:59:15.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: 11-20T08:59:14.595653+0000 2021-11-20T08:59:15.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: mon.smithi093 (mon.0) 681 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:59:15.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: audit 2021 2021-11-20T08:59:15.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: -11-20T08:59:14.595737+0000 2021-11-20T08:59:15.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: mon.smithi093 (mon.0) 682 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:59:15.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: 2021-11-20T08:59:15.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: cluster 2021-11-20T08:59:14. 2021-11-20T08:59:15.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: 598496+0000 mon.smithi093 (mon.0) 683 : cluster 2021-11-20T08:59:15.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:59:15.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T08:59:15.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: 08:59:14.994671+0000 mon.smithi093 (mon. 2021-11-20T08:59:15.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:15 smithi102 conmon[36472]: 0) 684 : audit [DBG] from='client.? 172.21.15.93:0/58297578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:15.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: cluster 2021-11-20T08:59:14.587567+0000 mon.smithi093 (mon.0) 677 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:15.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:59:15.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: 59:14.595341+0000 mon.smithi093 (mon.0) 678 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:boot 2021-11-20T08:59:15.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: cluster 2021-11-20T08:59:14.595381+0000 mon.smithi093 (mon.0) 679 : cluster [INF] Standby daemon mds.cephfs.smithi093.ljzugc assigned to filesystem cephfs as rank 0 2021-11-20T08:59:15.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:59:15.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: 11-20T08:59:14.595480+0000 mon.smithi093 (mon.0) 680 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:59:15.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:59:15.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: 11-20T08:59:14.595653+0000 2021-11-20T08:59:15.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: mon.smithi093 (mon.0) 681 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:59:15.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: 2021-11-20T08:59:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: audit 2021-11-20T08:59:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: 2021-11-20T08:59:14.595737+0000 mon.smithi093 2021-11-20T08:59:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: (mon.0) 682 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.zrryyo"}]: dispatch 2021-11-20T08:59:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: cluster 2021 2021-11-20T08:59:15.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: -11-20T08:59:14.598496+0000 2021-11-20T08:59:15.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: mon.smithi093 (mon.0) 683 : cluster [DBG] 2021-11-20T08:59:15.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 3 up:standby 2021-11-20T08:59:15.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: audit 2021-11-20T08:59: 2021-11-20T08:59:15.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: 14.994671+0000 mon.smithi093 (mon.0) 684 : 2021-11-20T08:59:15.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:15 smithi093 conmon[33522]: audit [DBG] from='client.? 172.21.15.93:0/58297578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:16.008 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:16.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.130+0000 7fb7fe18d700 1 -- 172.21.15.93:0/1239315018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f7870 msgr2=0x7fb7f80f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:16.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.130+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/1239315018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f7870 0x7fb7f80f7cd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb7e8004660 tx=0x7fb7e8009b30).stop 2021-11-20T08:59:16.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.130+0000 7fb7fe18d700 1 -- 172.21.15.93:0/1239315018 shutdown_connections 2021-11-20T08:59:16.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.130+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/1239315018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f7870 0x7fb7f80f7cd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:16.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.130+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/1239315018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f80f65d0 0x7fb7f80f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:16.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.130+0000 7fb7fe18d700 1 -- 172.21.15.93:0/1239315018 >> 172.21.15.93:0/1239315018 conn(0x7fb7f80f1d10 msgr2=0x7fb7f80f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:16.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 -- 172.21.15.93:0/1239315018 shutdown_connections 2021-11-20T08:59:16.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 -- 172.21.15.93:0/1239315018 wait complete. 2021-11-20T08:59:16.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 Processor -- start 2021-11-20T08:59:16.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 -- start start 2021-11-20T08:59:16.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f65d0 0x7fb7f8106d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:16.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f80f7870 0x7fb7f8107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:16.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8107820 con 0x7fb7f80f65d0 2021-11-20T08:59:16.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fe18d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8108030 con 0x7fb7f80f7870 2021-11-20T08:59:16.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fd18b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f65d0 0x7fb7f8106d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:16.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fd18b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f65d0 0x7fb7f8106d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43686/0 (socket says 172.21.15.93:43686) 2021-11-20T08:59:16.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fd18b700 1 -- 172.21.15.93:0/4125273529 learned_addr learned my addr 172.21.15.93:0/4125273529 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:16.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fc98a700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f80f7870 0x7fb7f8107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:16.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fd18b700 1 -- 172.21.15.93:0/4125273529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f80f7870 msgr2=0x7fb7f8107220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:16.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fd18b700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f80f7870 0x7fb7f8107220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:16.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.131+0000 7fb7fd18b700 1 -- 172.21.15.93:0/4125273529 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e8005040 con 0x7fb7f80f65d0 2021-11-20T08:59:16.137 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.132+0000 7fb7fd18b700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f65d0 0x7fb7f8106d30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fb7f4002700 tx=0x7fb7f4009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:16.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.132+0000 7fb7edffb700 1 -- 172.21.15.93:0/4125273529 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f4009320 con 0x7fb7f80f65d0 2021-11-20T08:59:16.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.132+0000 7fb7edffb700 1 -- 172.21.15.93:0/4125273529 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7f400f040 con 0x7fb7f80f65d0 2021-11-20T08:59:16.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.132+0000 7fb7edffb700 1 -- 172.21.15.93:0/4125273529 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f40148e0 con 0x7fb7f80f65d0 2021-11-20T08:59:16.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.132+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f81082b0 con 0x7fb7f80f65d0 2021-11-20T08:59:16.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.132+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f8108870 con 0x7fb7f80f65d0 2021-11-20T08:59:16.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.133+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f80f59c0 con 0x7fb7f80f65d0 2021-11-20T08:59:16.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.133+0000 7fb7edffb700 1 -- 172.21.15.93:0/4125273529 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb7f4009480 con 0x7fb7f80f65d0 2021-11-20T08:59:16.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.133+0000 7fb7edffb700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb7e405c300 0x7fb7e405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:16.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.133+0000 7fb7edffb700 1 -- 172.21.15.93:0/4125273529 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb7f40764b0 con 0x7fb7f80f65d0 2021-11-20T08:59:16.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.134+0000 7fb7fc98a700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb7e405c300 0x7fb7e405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:16.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.134+0000 7fb7fc98a700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb7e405c300 0x7fb7e405e7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb7e8004e30 tx=0x7fb7e8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:16.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.137+0000 7fb7edffb700 1 -- 172.21.15.93:0/4125273529 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7f4048e70 con 0x7fb7f80f65d0 2021-11-20T08:59:16.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.321+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7f80fee80 con 0x7fb7f80f65d0 2021-11-20T08:59:16.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.322+0000 7fb7edffb700 1 -- 172.21.15.93:0/4125273529 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2939 (secure 0 0 0) 0x7fb7f80fee80 con 0x7fb7f80f65d0 2021-11-20T08:59:16.321 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:16.322 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:15.596767+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:16.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb7e405c300 msgr2=0x7fb7e405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:16.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb7e405c300 0x7fb7e405e7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb7e8004e30 tx=0x7fb7e8009b30).stop 2021-11-20T08:59:16.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f65d0 msgr2=0x7fb7f8106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:16.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f65d0 0x7fb7f8106d30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fb7f4002700 tx=0x7fb7f4009e00).stop 2021-11-20T08:59:16.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 shutdown_connections 2021-11-20T08:59:16.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb7e405c300 0x7fb7e405e7c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:16.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7f80f65d0 0x7fb7f8106d30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:16.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 --2- 172.21.15.93:0/4125273529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7f80f7870 0x7fb7f8107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:16.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.323+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 >> 172.21.15.93:0/4125273529 conn(0x7fb7f80f1d10 msgr2=0x7fb7f80fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:16.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.324+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 shutdown_connections 2021-11-20T08:59:16.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:16.324+0000 7fb7fe18d700 1 -- 172.21.15.93:0/4125273529 wait complete. 2021-11-20T08:59:16.326 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 56 2021-11-20T08:59:16.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:15.596767+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:16.335 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:reconnect 2021-11-20T08:59:16.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: cluster 2021-11-20T08:59:15.558708+0000 mgr.smithi093.qdikkm (mgr.14162) 261 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:59:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: 2021-11-20T08:59:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: cluster 2021-11-20T08:59:15.605095+0000 mon.smithi093 (mon.0) 685 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:reconnect 2021-11-20T08:59:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: cluster 2021-11-20T08:59:15.605159+0000 mon.smithi093 (mon.0) 686 2021-11-20T08:59:16.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:59:16.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: cluster 2021-11-20T08:59: 2021-11-20T08:59:16.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: 15.624748+0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:59:16.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: audit 2021 2021-11-20T08:59:16.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: -11-20T08:59:16.322738+0000 2021-11-20T08:59:16.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: mon.smithi093 (mon.0) 687 : audit [DBG] 2021-11-20T08:59:16.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:16 smithi102 conmon[36472]: from='client.? 172.21.15.93:0/4125273529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:16.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: cluster 2021-11-20T08:59:15.558708+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:59:16.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: 261 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:59:16.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: cluster 2021-11-20T08:59:15.605095+0000 mon.smithi093 (mon.0) 685 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:reconnect 2021-11-20T08:59:16.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:59:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: 08:59:15.605159+0000 mon.smithi093 (mon.0) 686 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 3 up:standby 2021-11-20T08:59:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: cluster 2021-11-20T08:59:15.624748 2021-11-20T08:59:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: +0000 mds.cephfs.smithi093.ljzugc (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:59:16.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: audit 2021 2021-11-20T08:59:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: -11-20T08:59:16.322738+0000 mon.smithi093 (mon 2021-11-20T08:59:16.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:16 smithi093 conmon[33522]: .0) 687 : audit [DBG] from='client.? 172.21.15.93:0/4125273529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:17.335 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:17.464 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 -- 172.21.15.93:0/1651038466 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 msgr2=0x7fdd980f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:17.465 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1651038466 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd980f59c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fdd88004660 tx=0x7fdd88009b30).stop 2021-11-20T08:59:17.465 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 -- 172.21.15.93:0/1651038466 shutdown_connections 2021-11-20T08:59:17.465 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1651038466 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd980f6840 0x7fdd980f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:17.465 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1651038466 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd980f59c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:17.465 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 -- 172.21.15.93:0/1651038466 >> 172.21.15.93:0/1651038466 conn(0x7fdd980f0ce0 msgr2=0x7fdd980f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:17.466 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 -- 172.21.15.93:0/1651038466 shutdown_connections 2021-11-20T08:59:17.466 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 -- 172.21.15.93:0/1651038466 wait complete. 2021-11-20T08:59:17.466 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 Processor -- start 2021-11-20T08:59:17.466 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 -- start start 2021-11-20T08:59:17.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.460+0000 7fdd9db98700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd98106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:17.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd9db98700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd980f6840 0x7fdd981073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:17.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd9db98700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98107a00 con 0x7fdd980f55a0 2021-11-20T08:59:17.467 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd9db98700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98107b40 con 0x7fdd980f6840 2021-11-20T08:59:17.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd977fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd98106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:17.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd977fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd98106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43696/0 (socket says 172.21.15.93:43696) 2021-11-20T08:59:17.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd977fe700 1 -- 172.21.15.93:0/1603450607 learned_addr learned my addr 172.21.15.93:0/1603450607 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:17.468 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd977fe700 1 -- 172.21.15.93:0/1603450607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd980f6840 msgr2=0x7fdd981073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:17.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd977fe700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd980f6840 0x7fdd981073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:17.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd977fe700 1 -- 172.21.15.93:0/1603450607 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88005040 con 0x7fdd980f55a0 2021-11-20T08:59:17.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd977fe700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd98106ed0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fdd88007a70 tx=0x7fdd88009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:17.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd7ffff700 1 -- 172.21.15.93:0/1603450607 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd88018070 con 0x7fdd980f55a0 2021-11-20T08:59:17.469 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd7ffff700 1 -- 172.21.15.93:0/1603450607 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd88009700 con 0x7fdd980f55a0 2021-11-20T08:59:17.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9810c580 con 0x7fdd980f55a0 2021-11-20T08:59:17.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd7ffff700 1 -- 172.21.15.93:0/1603450607 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8801c620 con 0x7fdd980f55a0 2021-11-20T08:59:17.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.461+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9810cae0 con 0x7fdd980f55a0 2021-11-20T08:59:17.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.462+0000 7fdd7ffff700 1 -- 172.21.15.93:0/1603450607 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdd8801d420 con 0x7fdd980f55a0 2021-11-20T08:59:17.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.462+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd980fddb0 con 0x7fdd980f55a0 2021-11-20T08:59:17.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.463+0000 7fdd7ffff700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8005c290 0x7fdd8005e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:17.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.463+0000 7fdd7ffff700 1 -- 172.21.15.93:0/1603450607 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fdd88078120 con 0x7fdd980f55a0 2021-11-20T08:59:17.471 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.463+0000 7fdd96ffd700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8005c290 0x7fdd8005e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:17.472 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.463+0000 7fdd96ffd700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8005c290 0x7fdd8005e750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fdd8c009610 tx=0x7fdd8c006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:17.472 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.467+0000 7fdd7ffff700 1 -- 172.21.15.93:0/1603450607 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd8804aa80 con 0x7fdd980f55a0 2021-11-20T08:59:17.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.658+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd9810d120 con 0x7fdd980f55a0 2021-11-20T08:59:17.657 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.658+0000 7fdd7ffff700 1 -- 172.21.15.93:0/1603450607 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2942 (secure 0 0 0) 0x7fdd8804e0a0 con 0x7fdd980f55a0 2021-11-20T08:59:17.658 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:17.658 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:17.659 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8005c290 msgr2=0x7fdd8005e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:17.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8005c290 0x7fdd8005e750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fdd8c009610 tx=0x7fdd8c006ab0).stop 2021-11-20T08:59:17.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 msgr2=0x7fdd98106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:17.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd98106ed0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fdd88007a70 tx=0x7fdd88009b30).stop 2021-11-20T08:59:17.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 shutdown_connections 2021-11-20T08:59:17.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fdd8005c290 0x7fdd8005e750 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:17.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd980f55a0 0x7fdd98106ed0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:17.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 --2- 172.21.15.93:0/1603450607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd980f6840 0x7fdd981073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:17.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 >> 172.21.15.93:0/1603450607 conn(0x7fdd980f0ce0 msgr2=0x7fdd9805fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:17.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 shutdown_connections 2021-11-20T08:59:17.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:17.660+0000 7fdd9db98700 1 -- 172.21.15.93:0/1603450607 wait complete. 2021-11-20T08:59:17.662 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 58 2021-11-20T08:59:17.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:17.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:17.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:17 smithi102 conmon[36472]: cluster 2021-11-20T08:59:16.606105+0000 mon.smithi093 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:rejoin 2021-11-20T08:59:17.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:17 smithi102 conmon[36472]: cluster 2021-11-20T08:59:16. 2021-11-20T08:59:17.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:17 smithi102 conmon[36472]: 606154+0000 mon.smithi093 (mon.0) 689 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:59:17.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:17 smithi093 conmon[33522]: cluster 2021-11-20T08:59:16.606105+0000 mon.smithi093 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:rejoin 2021-11-20T08:59:17.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:17 smithi093 conmon[33522]: cluster 2021-11-20T08:59:16.606154+0000 mon.smithi093 (mon.0) 689 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 3 up:standby 2021-11-20T08:59:17.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:17 smithi093 conmon[33522]: 2021-11-20T08:59:18.672 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:18.797 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 -- 172.21.15.93:0/4113448828 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 msgr2=0x7f251c0f5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:18.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 --2- 172.21.15.93:0/4113448828 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c0f5e30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f2518004660 tx=0x7f2518009b30).stop 2021-11-20T08:59:18.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 -- 172.21.15.93:0/4113448828 shutdown_connections 2021-11-20T08:59:18.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 --2- 172.21.15.93:0/4113448828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251c0f4760 0x7f251c0f4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:18.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 --2- 172.21.15.93:0/4113448828 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c0f5e30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:18.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 -- 172.21.15.93:0/4113448828 >> 172.21.15.93:0/4113448828 conn(0x7f251c0f0440 msgr2=0x7f251c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:18.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 -- 172.21.15.93:0/4113448828 shutdown_connections 2021-11-20T08:59:18.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.798+0000 7f2523a78700 1 -- 172.21.15.93:0/4113448828 wait complete. 2021-11-20T08:59:18.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 Processor -- start 2021-11-20T08:59:18.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 -- start start 2021-11-20T08:59:18.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251c0f4760 0x7f251c10d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:18.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c10d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:18.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c10de90 con 0x7f251c0f5a10 2021-11-20T08:59:18.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c10dfd0 con 0x7f251c0f4760 2021-11-20T08:59:18.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521013700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c10d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:18.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521013700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c10d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43710/0 (socket says 172.21.15.93:43710) 2021-11-20T08:59:18.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521013700 1 -- 172.21.15.93:0/3655153050 learned_addr learned my addr 172.21.15.93:0/3655153050 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:18.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521814700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251c0f4760 0x7f251c10d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:18.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521013700 1 -- 172.21.15.93:0/3655153050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251c0f4760 msgr2=0x7f251c10d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:18.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521013700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251c0f4760 0x7f251c10d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:18.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521013700 1 -- 172.21.15.93:0/3655153050 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2518005040 con 0x7f251c0f5a10 2021-11-20T08:59:18.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2521013700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c10d8e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f250c002700 tx=0x7f250c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:18.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f25127fc700 1 -- 172.21.15.93:0/3655153050 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250c009320 con 0x7f251c0f5a10 2021-11-20T08:59:18.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f25127fc700 1 -- 172.21.15.93:0/3655153050 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f250c00f040 con 0x7f251c0f5a10 2021-11-20T08:59:18.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c112a70 con 0x7f251c0f5a10 2021-11-20T08:59:18.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f25127fc700 1 -- 172.21.15.93:0/3655153050 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250c014820 con 0x7f251c0f5a10 2021-11-20T08:59:18.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.799+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c113030 con 0x7f251c0f5a10 2021-11-20T08:59:18.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.800+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c027a00 con 0x7f251c0f5a10 2021-11-20T08:59:18.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.801+0000 7f25127fc700 1 -- 172.21.15.93:0/3655153050 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f250c0077b0 con 0x7f251c0f5a10 2021-11-20T08:59:18.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.801+0000 7f25127fc700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f250805c2a0 0x7f250805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:18.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.801+0000 7f25127fc700 1 -- 172.21.15.93:0/3655153050 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f250c0764d0 con 0x7f251c0f5a10 2021-11-20T08:59:18.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.801+0000 7f2521814700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f250805c2a0 0x7f250805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:18.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.802+0000 7f2521814700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f250805c2a0 0x7f250805e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2518004e30 tx=0x7f2518009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:18.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.805+0000 7f25127fc700 1 -- 172.21.15.93:0/3655153050 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f250c048e90 con 0x7f251c0f5a10 2021-11-20T08:59:18.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:18 smithi102 conmon[36472]: cluster 2021-11-20T08:59:17.559084+0000 mgr.smithi093.qdikkm (mgr.14162) 262 : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T08:59:18.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:18 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T08:59:18.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:18 smithi102 conmon[36472]: 08:59:17.613916+0000 mon.smithi093 (mon.0) 690 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:clientreplay 2021-11-20T08:59:18.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:18 smithi102 conmon[36472]: cluster 2021-11-20T08:59:17.613960+0000 mon.smithi093 (mon.0) 691 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:59:18.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:18 smithi102 conmon[36472]: audit 2021-11-20T08:59:18.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:18 smithi102 conmon[36472]: 2021-11-20T08:59:17.659198+0000 mon.smithi093 (mon.0) 692 : audit [DBG] from='client.? 172.21.15.93:0/1603450607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:18.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: cluster 2021-11-20T08:59:17.559084+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:59:18.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: .14162) 262 : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T08:59:18.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: cluster 2021-11-20T08:59:17.613916+0000 mon.smithi093 (mon.0) 690 : cluster [DBG] mds.? [v2:172.21.15.93:6826/1373400403,v1:172.21.15.93:6827/1373400403] up:clientreplay 2021-11-20T08:59:18.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: cluster 2021-11-20T08:59:17.613960+0000 mon.smithi093 (mon.0) 691 : 2021-11-20T08:59:18.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:59:18.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:59:18.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: 59:17.659198+0000 mon.smithi093 (mon.0) 692 : audit [DBG] from='client.? 172.21.15.93:0/1603450607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:18.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:18 smithi093 conmon[33522]: 2021-11-20T08:59:18.991 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.992+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f251c105480 con 0x7f251c0f5a10 2021-11-20T08:59:18.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.994+0000 7f25127fc700 1 -- 172.21.15.93:0/3655153050 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f250c04c4b0 con 0x7f251c0f5a10 2021-11-20T08:59:18.993 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:18.993 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:18.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f250805c2a0 msgr2=0x7f250805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:18.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f250805c2a0 0x7f250805e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2518004e30 tx=0x7f2518009b30).stop 2021-11-20T08:59:18.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 msgr2=0x7f251c10d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:18.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c10d8e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f250c002700 tx=0x7f250c009e00).stop 2021-11-20T08:59:18.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 shutdown_connections 2021-11-20T08:59:18.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f250805c2a0 0x7f250805e760 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:18.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251c0f4760 0x7f251c10d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:18.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 --2- 172.21.15.93:0/3655153050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f251c0f5a10 0x7f251c10d8e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:18.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 >> 172.21.15.93:0/3655153050 conn(0x7f251c0f0440 msgr2=0x7f251c0f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:18.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 shutdown_connections 2021-11-20T08:59:18.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:18.997+0000 7f2523a78700 1 -- 172.21.15.93:0/3655153050 wait complete. 2021-11-20T08:59:19.002 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:19.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:19.012 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:19.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:19 smithi102 conmon[36472]: cluster 2021-11-20T08:59:18.616085+0000 mon.smithi093 (mon.0) 693 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:standby 2021-11-20T08:59:19.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:19 smithi102 conmon[36472]: cluster 2021-11-20T08:59:18. 2021-11-20T08:59:19.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:19 smithi102 conmon[36472]: 616151+0000 mon.smithi093 (mon.0) 694 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:59:19.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:19 smithi102 conmon[36472]: audit 2021-11-20T08:59:18.994377+0000 mon.smithi093 (mon.0) 695 : audit [DBG] from='client.? 172.21.15.93:0/3655153050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:19.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:19 smithi093 conmon[33522]: cluster 2021-11-20T08:59:18.616085+0000 mon.smithi093 (mon.0) 2021-11-20T08:59:19.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:19 smithi093 conmon[33522]: 693 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:standby 2021-11-20T08:59:19.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:19 smithi093 conmon[33522]: cluster 2021-11-20T08:59:18.616151+0000 mon.smithi093 (mon.0) 694 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 3 up:standby 2021-11-20T08:59:19.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:19 smithi093 conmon[33522]: audit 2021-11-20T08:59:19.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:19 smithi093 conmon[33522]: 2021-11-20T08:59:18.994377+0000 mon.smithi093 (mon.0) 695 : audit [DBG] from='client.? 172.21.15.93:0/3655153050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:20.013 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:20.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 -- 172.21.15.93:0/48941255 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 msgr2=0x7fca2c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:20.143 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 --2- 172.21.15.93:0/48941255 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c0f6c80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fca1c004660 tx=0x7fca1c009b30).stop 2021-11-20T08:59:20.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 -- 172.21.15.93:0/48941255 shutdown_connections 2021-11-20T08:59:20.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 --2- 172.21.15.93:0/48941255 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c0f6c80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:20.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 --2- 172.21.15.93:0/48941255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0f5580 0x7fca2c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:20.144 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 -- 172.21.15.93:0/48941255 >> 172.21.15.93:0/48941255 conn(0x7fca2c0f0ce0 msgr2=0x7fca2c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:20.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 -- 172.21.15.93:0/48941255 shutdown_connections 2021-11-20T08:59:20.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 -- 172.21.15.93:0/48941255 wait complete. 2021-11-20T08:59:20.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.139+0000 7fca33acc700 1 Processor -- start 2021-11-20T08:59:20.145 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca33acc700 1 -- start start 2021-11-20T08:59:20.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca33acc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0f5580 0x7fca2c0604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:20.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca33acc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c10bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:20.146 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca33acc700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c10c120 con 0x7fca2c0f6820 2021-11-20T08:59:20.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca33acc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c10c260 con 0x7fca2c0f5580 2021-11-20T08:59:20.147 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31067700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c10bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:20.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31067700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c10bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43722/0 (socket says 172.21.15.93:43722) 2021-11-20T08:59:20.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31067700 1 -- 172.21.15.93:0/470883783 learned_addr learned my addr 172.21.15.93:0/470883783 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:20.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31868700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0f5580 0x7fca2c0604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:20.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31067700 1 -- 172.21.15.93:0/470883783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0f5580 msgr2=0x7fca2c0604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:20.148 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31067700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0f5580 0x7fca2c0604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:20.149 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31067700 1 -- 172.21.15.93:0/470883783 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c005040 con 0x7fca2c0f6820 2021-11-20T08:59:20.149 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca31067700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c10bc30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fca1c000c00 tx=0x7fca1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:20.150 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca227fc700 1 -- 172.21.15.93:0/470883783 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1c019070 con 0x7fca2c0f6820 2021-11-20T08:59:20.150 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca227fc700 1 -- 172.21.15.93:0/470883783 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca1c007e70 con 0x7fca2c0f6820 2021-11-20T08:59:20.150 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca227fc700 1 -- 172.21.15.93:0/470883783 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1c014d00 con 0x7fca2c0f6820 2021-11-20T08:59:20.150 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c10c460 con 0x7fca2c0f6820 2021-11-20T08:59:20.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.140+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c10c990 con 0x7fca2c0f6820 2021-11-20T08:59:20.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.141+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c046330 con 0x7fca2c0f6820 2021-11-20T08:59:20.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.142+0000 7fca227fc700 1 -- 172.21.15.93:0/470883783 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fca1c025070 con 0x7fca2c0f6820 2021-11-20T08:59:20.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.142+0000 7fca227fc700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fca1805c2a0 0x7fca1805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:20.152 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.142+0000 7fca227fc700 1 -- 172.21.15.93:0/470883783 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fca1c077120 con 0x7fca2c0f6820 2021-11-20T08:59:20.152 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.142+0000 7fca31868700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fca1805c2a0 0x7fca1805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:20.153 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.142+0000 7fca31868700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fca1805c2a0 0x7fca1805e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fca2c0f6680 tx=0x7fca2800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:20.153 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.146+0000 7fca227fc700 1 -- 172.21.15.93:0/470883783 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca1c049b00 con 0x7fca2c0f6820 2021-11-20T08:59:20.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.332+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca2c027a00 con 0x7fca2c0f6820 2021-11-20T08:59:20.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.333+0000 7fca227fc700 1 -- 172.21.15.93:0/470883783 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7fca1c023070 con 0x7fca2c0f6820 2021-11-20T08:59:20.332 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:20.332 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:20.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fca1805c2a0 msgr2=0x7fca1805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:20.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fca1805c2a0 0x7fca1805e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fca2c0f6680 tx=0x7fca2800b040).stop 2021-11-20T08:59:20.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 msgr2=0x7fca2c10bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:20.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c10bc30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fca1c000c00 tx=0x7fca1c00b040).stop 2021-11-20T08:59:20.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 shutdown_connections 2021-11-20T08:59:20.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fca1805c2a0 0x7fca1805e760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:20.336 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0f5580 0x7fca2c0604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:20.336 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 --2- 172.21.15.93:0/470883783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca2c0f6820 0x7fca2c10bc30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:20.336 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 >> 172.21.15.93:0/470883783 conn(0x7fca2c0f0ce0 msgr2=0x7fca2c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:20.336 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 shutdown_connections 2021-11-20T08:59:20.337 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:20.334+0000 7fca33acc700 1 -- 172.21.15.93:0/470883783 wait complete. 2021-11-20T08:59:20.337 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:20.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:20.345 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:20.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:20 smithi102 conmon[36472]: cluster 2021-11-20T08:59:19.559567+0000 mgr.smithi093.qdikkm (mgr.14162) 263 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:20.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:20 smithi102 conmon[36472]: audit 2021-11-20T08:59:20.333640+0000 mon.smithi093 (mon.0) 696 : audit [DBG] from='client.? 172.21.15.93:0/470883783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:20 smithi093 conmon[33522]: cluster 2021-11-20T08:59:19.559567+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:59:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:20 smithi093 conmon[33522]: 14162) 263 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 2.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T08:59:20.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:20 smithi093 conmon[33522]: audit 2021-11-20T08:59:20.333640+0000 mon.smithi093 (mon.0) 696 : audit [DBG] from='client.? 172.21.15.93:0/470883783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:21.346 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:21.481 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 -- 172.21.15.93:0/3033927334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f6680 msgr2=0x7f12bc0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:21.482 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/3033927334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f6680 0x7f12bc0f6aa0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f12ac004660 tx=0x7f12ac009b30).stop 2021-11-20T08:59:21.482 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 -- 172.21.15.93:0/3033927334 shutdown_connections 2021-11-20T08:59:21.482 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/3033927334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12bc0f7920 0x7f12bc0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:21.482 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/3033927334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f6680 0x7f12bc0f6aa0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:21.483 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 -- 172.21.15.93:0/3033927334 >> 172.21.15.93:0/3033927334 conn(0x7f12bc0f1de0 msgr2=0x7f12bc0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:21.483 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 -- 172.21.15.93:0/3033927334 shutdown_connections 2021-11-20T08:59:21.483 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.480+0000 7f12c2dc8700 1 -- 172.21.15.93:0/3033927334 wait complete. 2021-11-20T08:59:21.483 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12c2dc8700 1 Processor -- start 2021-11-20T08:59:21.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12c2dc8700 1 -- start start 2021-11-20T08:59:21.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12c2dc8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12bc0f6680 0x7f12bc06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:21.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12c2dc8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f7920 0x7f12bc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:21.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12c2dc8700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc06bd70 con 0x7f12bc0f7920 2021-11-20T08:59:21.484 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12c2dc8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc06a310 con 0x7f12bc0f6680 2021-11-20T08:59:21.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12bbfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f7920 0x7f12bc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:21.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12bbfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f7920 0x7f12bc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43734/0 (socket says 172.21.15.93:43734) 2021-11-20T08:59:21.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12bbfff700 1 -- 172.21.15.93:0/1457165603 learned_addr learned my addr 172.21.15.93:0/1457165603 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:21.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.481+0000 7f12c0b64700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12bc0f6680 0x7f12bc06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:21.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12bbfff700 1 -- 172.21.15.93:0/1457165603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12bc0f6680 msgr2=0x7f12bc06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:21.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12bbfff700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12bc0f6680 0x7f12bc06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:21.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12bbfff700 1 -- 172.21.15.93:0/1457165603 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac005040 con 0x7f12bc0f7920 2021-11-20T08:59:21.486 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12bbfff700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f7920 0x7f12bc069e20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f12a400eb70 tx=0x7f12a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:21.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12b97fa700 1 -- 172.21.15.93:0/1457165603 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a40092e0 con 0x7f12bc0f7920 2021-11-20T08:59:21.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12b97fa700 1 -- 172.21.15.93:0/1457165603 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12a400f040 con 0x7f12bc0f7920 2021-11-20T08:59:21.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12bc06a590 con 0x7f12bc0f7920 2021-11-20T08:59:21.487 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12b97fa700 1 -- 172.21.15.93:0/1457165603 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a400c910 con 0x7f12bc0f7920 2021-11-20T08:59:21.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.482+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc06ab50 con 0x7f12bc0f7920 2021-11-20T08:59:21.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.484+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc027a00 con 0x7f12bc0f7920 2021-11-20T08:59:21.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.484+0000 7f12b97fa700 1 -- 172.21.15.93:0/1457165603 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f12a401b030 con 0x7f12bc0f7920 2021-11-20T08:59:21.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.484+0000 7f12b97fa700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f12a805c2a0 0x7f12a805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:21.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.484+0000 7f12b97fa700 1 -- 172.21.15.93:0/1457165603 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f12a40760d0 con 0x7f12bc0f7920 2021-11-20T08:59:21.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.484+0000 7f12c0b64700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f12a805c2a0 0x7f12a805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:21.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.484+0000 7f12c0b64700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f12a805c2a0 0x7f12a805e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f12ac013530 tx=0x7f12ac016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:21.491 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.488+0000 7f12b97fa700 1 -- 172.21.15.93:0/1457165603 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12a4048bc0 con 0x7f12bc0f7920 2021-11-20T08:59:21.674 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.675+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12bc0fee10 con 0x7f12bc0f7920 2021-11-20T08:59:21.675 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.676+0000 7f12b97fa700 1 -- 172.21.15.93:0/1457165603 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f12a4019090 con 0x7f12bc0f7920 2021-11-20T08:59:21.675 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:21.676 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:21.677 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f12a805c2a0 msgr2=0x7f12a805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:21.677 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f12a805c2a0 0x7f12a805e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f12ac013530 tx=0x7f12ac016040).stop 2021-11-20T08:59:21.677 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f7920 msgr2=0x7f12bc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:21.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f7920 0x7f12bc069e20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f12a400eb70 tx=0x7f12a400b040).stop 2021-11-20T08:59:21.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 shutdown_connections 2021-11-20T08:59:21.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f12a805c2a0 0x7f12a805e760 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:21.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12bc0f6680 0x7f12bc06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:21.679 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 --2- 172.21.15.93:0/1457165603 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12bc0f7920 0x7f12bc069e20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:21.679 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.677+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 >> 172.21.15.93:0/1457165603 conn(0x7f12bc0f1de0 msgr2=0x7f12bc0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:21.679 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.678+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 shutdown_connections 2021-11-20T08:59:21.679 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:21.678+0000 7f12c2dc8700 1 -- 172.21.15.93:0/1457165603 wait complete. 2021-11-20T08:59:21.680 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:21.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:21.691 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:22.692 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:22.820 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/213893259 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f6820 msgr2=0x7fd5d80f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:22.820 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/213893259 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f6820 0x7fd5d80f6c80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fd5c8004660 tx=0x7fd5c8009b30).stop 2021-11-20T08:59:22.821 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/213893259 shutdown_connections 2021-11-20T08:59:22.821 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/213893259 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f6820 0x7fd5d80f6c80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:22.821 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/213893259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5d80f5580 0x7fd5d80f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:22.821 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/213893259 >> 172.21.15.93:0/213893259 conn(0x7fd5d80f0ce0 msgr2=0x7fd5d80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:22.822 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/213893259 shutdown_connections 2021-11-20T08:59:22.822 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.819+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/213893259 wait complete. 2021-11-20T08:59:22.822 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dfa4e700 1 Processor -- start 2021-11-20T08:59:22.822 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dfa4e700 1 -- start start 2021-11-20T08:59:22.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dfa4e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f5580 0x7fd5d8104ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:22.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dfa4e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5d80f6820 0x7fd5d8105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:22.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dfa4e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d8105790 con 0x7fd5d80f5580 2021-11-20T08:59:22.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dfa4e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d8105900 con 0x7fd5d80f6820 2021-11-20T08:59:22.824 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dd7ea700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f5580 0x7fd5d8104ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:22.824 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dd7ea700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f5580 0x7fd5d8104ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43744/0 (socket says 172.21.15.93:43744) 2021-11-20T08:59:22.824 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dd7ea700 1 -- 172.21.15.93:0/271517895 learned_addr learned my addr 172.21.15.93:0/271517895 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:22.824 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dcfe9700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5d80f6820 0x7fd5d8105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:22.824 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dd7ea700 1 -- 172.21.15.93:0/271517895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5d80f6820 msgr2=0x7fd5d8105190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:22.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dd7ea700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5d80f6820 0x7fd5d8105190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:22.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dd7ea700 1 -- 172.21.15.93:0/271517895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c8005040 con 0x7fd5d80f5580 2021-11-20T08:59:22.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.820+0000 7fd5dcfe9700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5d80f6820 0x7fd5d8105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:59:22.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.821+0000 7fd5dd7ea700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f5580 0x7fd5d8104ca0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fd5d4002850 tx=0x7fd5d400bf90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:22.826 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.821+0000 7fd5cdffb700 1 -- 172.21.15.93:0/271517895 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5d4002df0 con 0x7fd5d80f5580 2021-11-20T08:59:22.826 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.821+0000 7fd5cdffb700 1 -- 172.21.15.93:0/271517895 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5d4011040 con 0x7fd5d80f5580 2021-11-20T08:59:22.826 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.821+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d810a370 con 0x7fd5d80f5580 2021-11-20T08:59:22.827 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.821+0000 7fd5cdffb700 1 -- 172.21.15.93:0/271517895 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5d400f8b0 con 0x7fd5d80f5580 2021-11-20T08:59:22.827 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.821+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d810a900 con 0x7fd5d80f5580 2021-11-20T08:59:22.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.822+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d8046330 con 0x7fd5d80f5580 2021-11-20T08:59:22.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.822+0000 7fd5cdffb700 1 -- 172.21.15.93:0/271517895 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd5d400fae0 con 0x7fd5d80f5580 2021-11-20T08:59:22.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.822+0000 7fd5cdffb700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd5c405c2a0 0x7fd5c405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:22.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.823+0000 7fd5cdffb700 1 -- 172.21.15.93:0/271517895 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd5d4078640 con 0x7fd5d80f5580 2021-11-20T08:59:22.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.823+0000 7fd5dcfe9700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd5c405c2a0 0x7fd5c405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:22.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.823+0000 7fd5dcfe9700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd5c405c2a0 0x7fd5c405e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd5c8004e30 tx=0x7fd5c8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:22.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:22.826+0000 7fd5cdffb700 1 -- 172.21.15.93:0/271517895 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5d404b000 con 0x7fd5d80f5580 2021-11-20T08:59:22.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:22 smithi102 conmon[36472]: cluster 2021-11-20T08:59:21.559955+0000 mgr.smithi093.qdikkm (mgr.14162) 264 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:22.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:22 smithi102 conmon[36472]: audit 2021-11-20T08:59:21.676743+0000 mon.smithi093 (mon.0) 697 : audit [DBG] from='client.? 172.21.15.93:0/1457165603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:22 smithi093 conmon[33522]: cluster 2021-11-20T08:59:21.559955+0000 mgr.smithi093.qdikkm (mgr.14162) 264 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:22 smithi093 conmon[33522]: 2021-11-20T08:59:22.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:22 smithi093 conmon[33522]: audit 2021-11-20T08:59:21.676743+0000 mon.smithi093 (mon.0) 697 : audit [DBG] from='client.? 172.21.15.93:0/1457165603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:23.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.010+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5d8027a00 con 0x7fd5d80f5580 2021-11-20T08:59:23.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.011+0000 7fd5cdffb700 1 -- 172.21.15.93:0/271517895 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7fd5d404e620 con 0x7fd5d80f5580 2021-11-20T08:59:23.010 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:23.011 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:23.012 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd5c405c2a0 msgr2=0x7fd5c405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:23.013 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd5c405c2a0 0x7fd5c405e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd5c8004e30 tx=0x7fd5c8009b30).stop 2021-11-20T08:59:23.013 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f5580 msgr2=0x7fd5d8104ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:23.013 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f5580 0x7fd5d8104ca0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fd5d4002850 tx=0x7fd5d400bf90).stop 2021-11-20T08:59:23.013 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 shutdown_connections 2021-11-20T08:59:23.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd5c405c2a0 0x7fd5c405e760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:23.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5d80f5580 0x7fd5d8104ca0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:23.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 --2- 172.21.15.93:0/271517895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5d80f6820 0x7fd5d8105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:23.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 >> 172.21.15.93:0/271517895 conn(0x7fd5d80f0ce0 msgr2=0x7fd5d80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:23.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 shutdown_connections 2021-11-20T08:59:23.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:23.013+0000 7fd5dfa4e700 1 -- 172.21.15.93:0/271517895 wait complete. 2021-11-20T08:59:23.015 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:23.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:23.024 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:23.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:23 smithi102 conmon[36472]: audit 2021-11-20T08:59:23.012028+0000 mon.smithi093 (mon.0) 698 : audit [DBG] from='client.? 172.21.15.93:0/271517895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:23.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:23 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T08:59:23.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:23 smithi093 conmon[33522]: 20T08:59:23.012028+0000 mon.smithi093 (mon.0) 698 : audit [DBG] from='client.? 172.21.15.93:0/271517895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:24.024 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:24.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.149+0000 7f92c0935700 1 -- 172.21.15.93:0/1245322484 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 msgr2=0x7f92bc0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:24.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.149+0000 7f92c0935700 1 --2- 172.21.15.93:0/1245322484 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f59a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f92ac004660 tx=0x7f92ac009b30).stop 2021-11-20T08:59:24.151 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 -- 172.21.15.93:0/1245322484 shutdown_connections 2021-11-20T08:59:24.152 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 --2- 172.21.15.93:0/1245322484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92bc0f6820 0x7f92bc0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:24.152 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 --2- 172.21.15.93:0/1245322484 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f59a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:24.152 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 -- 172.21.15.93:0/1245322484 >> 172.21.15.93:0/1245322484 conn(0x7f92bc0f0ce0 msgr2=0x7f92bc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:24.152 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 -- 172.21.15.93:0/1245322484 shutdown_connections 2021-11-20T08:59:24.153 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 -- 172.21.15.93:0/1245322484 wait complete. 2021-11-20T08:59:24.153 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 Processor -- start 2021-11-20T08:59:24.153 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 -- start start 2021-11-20T08:59:24.153 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:24.154 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92bc0f6820 0x7f92bc0f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:24.154 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0fa2d0 con 0x7f92bc0f5580 2021-11-20T08:59:24.154 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92c0935700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0f8870 con 0x7f92bc0f6820 2021-11-20T08:59:24.154 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92ba59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:24.155 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92ba59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43754/0 (socket says 172.21.15.93:43754) 2021-11-20T08:59:24.155 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92ba59c700 1 -- 172.21.15.93:0/2011483343 learned_addr learned my addr 172.21.15.93:0/2011483343 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:24.155 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.150+0000 7f92b9d9b700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92bc0f6820 0x7f92bc0f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:24.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92ba59c700 1 -- 172.21.15.93:0/2011483343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92bc0f6820 msgr2=0x7f92bc0f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:24.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92ba59c700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92bc0f6820 0x7f92bc0f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:24.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92ba59c700 1 -- 172.21.15.93:0/2011483343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac005040 con 0x7f92bc0f5580 2021-11-20T08:59:24.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92ba59c700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f9d20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f92ac009f50 tx=0x7f92ac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:24.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92aaffd700 1 -- 172.21.15.93:0/2011483343 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac018070 con 0x7f92bc0f5580 2021-11-20T08:59:24.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92aaffd700 1 -- 172.21.15.93:0/2011483343 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92ac013410 con 0x7f92bc0f5580 2021-11-20T08:59:24.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc0f8a90 con 0x7f92bc0f5580 2021-11-20T08:59:24.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92aaffd700 1 -- 172.21.15.93:0/2011483343 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac01c3c0 con 0x7f92bc0f5580 2021-11-20T08:59:24.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.151+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc0f8ff0 con 0x7f92bc0f5580 2021-11-20T08:59:24.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.152+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc046330 con 0x7f92bc0f5580 2021-11-20T08:59:24.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.152+0000 7f92aaffd700 1 -- 172.21.15.93:0/2011483343 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f92ac01d420 con 0x7f92bc0f5580 2021-11-20T08:59:24.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.153+0000 7f92aaffd700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f92a405c2a0 0x7f92a405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:24.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.153+0000 7f92aaffd700 1 -- 172.21.15.93:0/2011483343 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f92ac077f20 con 0x7f92bc0f5580 2021-11-20T08:59:24.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.153+0000 7f92b9d9b700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f92a405c2a0 0x7f92a405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:24.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.153+0000 7f92b9d9b700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f92a405c2a0 0x7f92a405e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f92b00041d0 tx=0x7f92b0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:24.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.156+0000 7f92aaffd700 1 -- 172.21.15.93:0/2011483343 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92ac0a42b0 con 0x7f92bc0f5580 2021-11-20T08:59:24.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.348+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92bc0fdd10 con 0x7f92bc0f5580 2021-11-20T08:59:24.347 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.348+0000 7f92aaffd700 1 -- 172.21.15.93:0/2011483343 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f92ac04a8e0 con 0x7f92bc0f5580 2021-11-20T08:59:24.348 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:24.348 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:24.349 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f92a405c2a0 msgr2=0x7f92a405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:24.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f92a405c2a0 0x7f92a405e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f92b00041d0 tx=0x7f92b0009040).stop 2021-11-20T08:59:24.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 msgr2=0x7f92bc0f9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:24.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f9d20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f92ac009f50 tx=0x7f92ac009b30).stop 2021-11-20T08:59:24.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 shutdown_connections 2021-11-20T08:59:24.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f92a405c2a0 0x7f92a405e760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:24.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92bc0f5580 0x7f92bc0f9d20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:24.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 --2- 172.21.15.93:0/2011483343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92bc0f6820 0x7f92bc0f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:24.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 >> 172.21.15.93:0/2011483343 conn(0x7f92bc0f0ce0 msgr2=0x7f92bc05fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:24.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 shutdown_connections 2021-11-20T08:59:24.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:24.350+0000 7f92c0935700 1 -- 172.21.15.93:0/2011483343 wait complete. 2021-11-20T08:59:24.352 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:24.361 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:24.361 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:24.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:24 smithi102 conmon[36472]: cluster 2021-11-20T08:59:23.560510+0000 mgr.smithi093.qdikkm (mgr.14162) 265 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:24.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:24 smithi102 conmon[36472]: audit 2021-11-20T08:59:24.349369+0000 mon.smithi093 (mon.0) 699 : audit [DBG] from='client.? 172.21.15.93:0/2011483343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:24 smithi093 conmon[33522]: cluster 2021-11-20T08:59:23.560510+0000 mgr.smithi093.qdikkm (mgr.14162) 265 2021-11-20T08:59:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:24 smithi093 conmon[33522]: : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:24.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:24 smithi093 conmon[33522]: audit 2021-11-20T08:59:24.349369+0000 mon.smithi093 (mon.0) 699 : audit [DBG] from='client.? 172.21.15.93:0/2011483343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:25.362 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:25.493 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 -- 172.21.15.93:0/1623409930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 msgr2=0x7f7b7c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:25.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 --2- 172.21.15.93:0/1623409930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c0f6c80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f7b64004660 tx=0x7f7b64009b30).stop 2021-11-20T08:59:25.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 -- 172.21.15.93:0/1623409930 shutdown_connections 2021-11-20T08:59:25.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 --2- 172.21.15.93:0/1623409930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c0f6c80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:25.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 --2- 172.21.15.93:0/1623409930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0f5580 0x7f7b7c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:25.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 -- 172.21.15.93:0/1623409930 >> 172.21.15.93:0/1623409930 conn(0x7f7b7c0f0ce0 msgr2=0x7f7b7c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:25.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 -- 172.21.15.93:0/1623409930 shutdown_connections 2021-11-20T08:59:25.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 -- 172.21.15.93:0/1623409930 wait complete. 2021-11-20T08:59:25.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.488+0000 7f7b8192e700 1 Processor -- start 2021-11-20T08:59:25.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b8192e700 1 -- start start 2021-11-20T08:59:25.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b8192e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0f5580 0x7f7b7c106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:25.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b8192e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:25.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b8192e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c107940 con 0x7f7b7c0f6820 2021-11-20T08:59:25.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b8192e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c107a80 con 0x7f7b7c0f5580 2021-11-20T08:59:25.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b727fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:25.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b727fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43766/0 (socket says 172.21.15.93:43766) 2021-11-20T08:59:25.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b727fc700 1 -- 172.21.15.93:0/4033374816 learned_addr learned my addr 172.21.15.93:0/4033374816 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:25.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b7affd700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0f5580 0x7f7b7c106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:25.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b727fc700 1 -- 172.21.15.93:0/4033374816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0f5580 msgr2=0x7f7b7c106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:25.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b727fc700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0f5580 0x7f7b7c106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:25.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b727fc700 1 -- 172.21.15.93:0/4033374816 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64005040 con 0x7f7b7c0f6820 2021-11-20T08:59:25.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b7affd700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0f5580 0x7f7b7c106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:59:25.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b727fc700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c107390 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7b64009f50 tx=0x7f7b64009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:25.507 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b73fff700 1 -- 172.21.15.93:0/4033374816 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b64018070 con 0x7f7b7c0f6820 2021-11-20T08:59:25.508 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b73fff700 1 -- 172.21.15.93:0/4033374816 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b64013410 con 0x7f7b7c0f6820 2021-11-20T08:59:25.508 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.489+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c10c4c0 con 0x7f7b7c0f6820 2021-11-20T08:59:25.508 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.490+0000 7f7b73fff700 1 -- 172.21.15.93:0/4033374816 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6401c3c0 con 0x7f7b7c0f6820 2021-11-20T08:59:25.508 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.490+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c10ca80 con 0x7f7b7c0f6820 2021-11-20T08:59:25.509 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.490+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c0fdd10 con 0x7f7b7c0f6820 2021-11-20T08:59:25.509 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.491+0000 7f7b73fff700 1 -- 172.21.15.93:0/4033374816 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7b6401d420 con 0x7f7b7c0f6820 2021-11-20T08:59:25.509 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.491+0000 7f7b73fff700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7b6805c2a0 0x7f7b6805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:25.509 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.491+0000 7f7b73fff700 1 -- 172.21.15.93:0/4033374816 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7b64077de0 con 0x7f7b7c0f6820 2021-11-20T08:59:25.510 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.491+0000 7f7b7affd700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7b6805c2a0 0x7f7b6805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:25.510 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.491+0000 7f7b7affd700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7b6805c2a0 0x7f7b6805e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7b7c0f6680 tx=0x7f7b6c009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:25.510 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.495+0000 7f7b73fff700 1 -- 172.21.15.93:0/4033374816 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b6404a820 con 0x7f7b7c0f6820 2021-11-20T08:59:25.679 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.681+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b7c10d110 con 0x7f7b7c0f6820 2021-11-20T08:59:25.680 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.681+0000 7f7b73fff700 1 -- 172.21.15.93:0/4033374816 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f7b6404de40 con 0x7f7b7c0f6820 2021-11-20T08:59:25.681 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:25.681 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:25.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7b6805c2a0 msgr2=0x7f7b6805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:25.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7b6805c2a0 0x7f7b6805e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7b7c0f6680 tx=0x7f7b6c009c00).stop 2021-11-20T08:59:25.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 msgr2=0x7f7b7c107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:25.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c107390 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7b64009f50 tx=0x7f7b64009b30).stop 2021-11-20T08:59:25.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 shutdown_connections 2021-11-20T08:59:25.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7b6805c2a0 0x7f7b6805e760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:25.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0f5580 0x7f7b7c106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:25.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 --2- 172.21.15.93:0/4033374816 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7b7c0f6820 0x7f7b7c107390 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:25.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 >> 172.21.15.93:0/4033374816 conn(0x7f7b7c0f0ce0 msgr2=0x7f7b7c05fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:25.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 shutdown_connections 2021-11-20T08:59:25.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:25.683+0000 7f7b8192e700 1 -- 172.21.15.93:0/4033374816 wait complete. 2021-11-20T08:59:25.685 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:25.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:25.694 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:26.695 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:26.828 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.826+0000 7fedc1f65700 1 -- 172.21.15.93:0/2223342793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f6760 msgr2=0x7fedbc0f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:26.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.826+0000 7fedc1f65700 1 --2- 172.21.15.93:0/2223342793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f6760 0x7fedbc0f6bc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7feda8004660 tx=0x7feda8009b30).stop 2021-11-20T08:59:26.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.826+0000 7fedc1f65700 1 -- 172.21.15.93:0/2223342793 shutdown_connections 2021-11-20T08:59:26.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.826+0000 7fedc1f65700 1 --2- 172.21.15.93:0/2223342793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f6760 0x7fedbc0f6bc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:26.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.826+0000 7fedc1f65700 1 --2- 172.21.15.93:0/2223342793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedbc0f54c0 0x7fedbc0f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:26.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.826+0000 7fedc1f65700 1 -- 172.21.15.93:0/2223342793 >> 172.21.15.93:0/2223342793 conn(0x7fedbc0f0c00 msgr2=0x7fedbc0f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:26.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 -- 172.21.15.93:0/2223342793 shutdown_connections 2021-11-20T08:59:26.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 -- 172.21.15.93:0/2223342793 wait complete. 2021-11-20T08:59:26.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 Processor -- start 2021-11-20T08:59:26.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 -- start start 2021-11-20T08:59:26.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f54c0 0x7fedbc109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:26.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedbc0f6760 0x7fedbc10bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:26.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc060560 con 0x7fedbc0f54c0 2021-11-20T08:59:26.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc1f65700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0606a0 con 0x7fedbc0f6760 2021-11-20T08:59:26.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.827+0000 7fedc0f63700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f54c0 0x7fedbc109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:26.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc0f63700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f54c0 0x7fedbc109a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43774/0 (socket says 172.21.15.93:43774) 2021-11-20T08:59:26.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc0f63700 1 -- 172.21.15.93:0/307329666 learned_addr learned my addr 172.21.15.93:0/307329666 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:26.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedbbfff700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedbc0f6760 0x7fedbc10bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:26.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc0f63700 1 -- 172.21.15.93:0/307329666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedbc0f6760 msgr2=0x7fedbc10bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:26.835 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc0f63700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedbc0f6760 0x7fedbc10bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:26.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc0f63700 1 -- 172.21.15.93:0/307329666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda8005040 con 0x7fedbc0f54c0 2021-11-20T08:59:26.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc0f63700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f54c0 0x7fedbc109a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fedb0002700 tx=0x7fedb0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:26.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedb97fa700 1 -- 172.21.15.93:0/307329666 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedb00079c0 con 0x7fedbc0f54c0 2021-11-20T08:59:26.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedb97fa700 1 -- 172.21.15.93:0/307329666 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedb000f040 con 0x7fedbc0f54c0 2021-11-20T08:59:26.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedb97fa700 1 -- 172.21.15.93:0/307329666 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedb0014860 con 0x7fedbc0f54c0 2021-11-20T08:59:26.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc10c510 con 0x7fedbc0f54c0 2021-11-20T08:59:26.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.828+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc10caa0 con 0x7fedbc0f54c0 2021-11-20T08:59:26.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.829+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedbc0f48b0 con 0x7fedbc0f54c0 2021-11-20T08:59:26.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.829+0000 7fedb97fa700 1 -- 172.21.15.93:0/307329666 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fedb001b060 con 0x7fedbc0f54c0 2021-11-20T08:59:26.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.830+0000 7fedb97fa700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fedac05c300 0x7fedac05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:26.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.830+0000 7fedb97fa700 1 -- 172.21.15.93:0/307329666 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fedb0076000 con 0x7fedbc0f54c0 2021-11-20T08:59:26.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.830+0000 7fedbbfff700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fedac05c300 0x7fedac05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:26.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.830+0000 7fedbbfff700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fedac05c300 0x7fedac05e7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7feda8009f50 tx=0x7feda8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:26.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:26.834+0000 7fedb97fa700 1 -- 172.21.15.93:0/307329666 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedb00489c0 con 0x7fedbc0f54c0 2021-11-20T08:59:26.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:26 smithi102 conmon[36472]: cluster 2021-11-20T08:59:25.560939+0000 mgr.smithi093.qdikkm (mgr.14162) 266 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 179 B/s wr, 1 op/s 2021-11-20T08:59:26.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:26 smithi102 conmon[36472]: 2021-11-20T08:59:26.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:26 smithi102 conmon[36472]: audit 2021-11-20T08:59:25.682192+0000 mon.smithi093 (mon.0) 700 : audit [DBG] from='client.? 172.21.15.93:0/4033374816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:26.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:26 smithi093 conmon[33522]: cluster 2021-11-20T08:59:25.560939+0000 mgr.smithi093.qdikkm (mgr.14162) 266 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 179 B/s wr, 1 op/s 2021-11-20T08:59:26.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:26 smithi093 conmon[33522]: 2021-11-20T08:59:26.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:26 smithi093 conmon[33522]: audit 2021-11-20T08:59:25.682192+0000 mon.smithi093 (mon.0) 700 : audit [DBG] from='client.? 172.21.15.93:0/4033374816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:27.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.018+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedbc045ad0 con 0x7fedbc0f54c0 2021-11-20T08:59:27.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.019+0000 7fedb97fa700 1 -- 172.21.15.93:0/307329666 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7fedb004bfe0 con 0x7fedbc0f54c0 2021-11-20T08:59:27.020 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:27.020 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:27.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fedac05c300 msgr2=0x7fedac05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fedac05c300 0x7fedac05e7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7feda8009f50 tx=0x7feda8009b30).stop 2021-11-20T08:59:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f54c0 msgr2=0x7fedbc109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f54c0 0x7fedbc109a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fedb0002700 tx=0x7fedb0009d80).stop 2021-11-20T08:59:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 shutdown_connections 2021-11-20T08:59:27.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fedac05c300 0x7fedac05e7c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:27.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedbc0f54c0 0x7fedbc109a60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:27.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 --2- 172.21.15.93:0/307329666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedbc0f6760 0x7fedbc10bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:27.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.020+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 >> 172.21.15.93:0/307329666 conn(0x7fedbc0f0c00 msgr2=0x7fedbc0fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:27.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.021+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 shutdown_connections 2021-11-20T08:59:27.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:27.021+0000 7fedc1f65700 1 -- 172.21.15.93:0/307329666 wait complete. 2021-11-20T08:59:27.025 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:27.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:27.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:27.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:27 smithi102 conmon[36472]: audit 2021-11-20T08:59:27.020783+0000 mon.smithi093 (mon.0) 2021-11-20T08:59:27.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:27 smithi102 conmon[36472]: 701 : audit [DBG] from='client.? 172.21.15.93:0/307329666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:27.941 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:27 smithi093 conmon[33522]: audit 2021-11-20T08:59:27.020783+0000 mon.smithi093 (mon.0) 701 : audit 2021-11-20T08:59:27.941 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:27 smithi093 conmon[33522]: [DBG] from='client.? 172.21.15.93:0/307329666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:28.034 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:28.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 -- 172.21.15.93:0/1452154002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f5490 msgr2=0x7fce840f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:28.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 --2- 172.21.15.93:0/1452154002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f5490 0x7fce840f58b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fce74004660 tx=0x7fce74009b30).stop 2021-11-20T08:59:28.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 -- 172.21.15.93:0/1452154002 shutdown_connections 2021-11-20T08:59:28.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 --2- 172.21.15.93:0/1452154002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce840f6730 0x7fce840f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:28.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 --2- 172.21.15.93:0/1452154002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f5490 0x7fce840f58b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:28.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 -- 172.21.15.93:0/1452154002 >> 172.21.15.93:0/1452154002 conn(0x7fce840f0bd0 msgr2=0x7fce840f3010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:28.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 -- 172.21.15.93:0/1452154002 shutdown_connections 2021-11-20T08:59:28.157 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.153+0000 7fce89759700 1 -- 172.21.15.93:0/1452154002 wait complete. 2021-11-20T08:59:28.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce89759700 1 Processor -- start 2021-11-20T08:59:28.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce89759700 1 -- start start 2021-11-20T08:59:28.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce89759700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce840f5490 0x7fce84109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:28.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce89759700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f6730 0x7fce8410bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:28.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce89759700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce840601f0 con 0x7fce840f6730 2021-11-20T08:59:28.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce89759700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84060330 con 0x7fce840f5490 2021-11-20T08:59:28.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce827fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f6730 0x7fce8410bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:28.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce827fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f6730 0x7fce8410bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43786/0 (socket says 172.21.15.93:43786) 2021-11-20T08:59:28.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce827fc700 1 -- 172.21.15.93:0/4217031 learned_addr learned my addr 172.21.15.93:0/4217031 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:28.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.154+0000 7fce82ffd700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce840f5490 0x7fce84109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:28.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce827fc700 1 -- 172.21.15.93:0/4217031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce840f5490 msgr2=0x7fce84109aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:28.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce827fc700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce840f5490 0x7fce84109aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:28.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce827fc700 1 -- 172.21.15.93:0/4217031 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74005040 con 0x7fce840f6730 2021-11-20T08:59:28.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce827fc700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f6730 0x7fce8410bfa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fce780026d0 tx=0x7fce78009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:28.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce6b7fe700 1 -- 172.21.15.93:0/4217031 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce780079c0 con 0x7fce840f6730 2021-11-20T08:59:28.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce6b7fe700 1 -- 172.21.15.93:0/4217031 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce7800f040 con 0x7fce840f6730 2021-11-20T08:59:28.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8410c550 con 0x7fce840f6730 2021-11-20T08:59:28.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce6b7fe700 1 -- 172.21.15.93:0/4217031 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce780148b0 con 0x7fce840f6730 2021-11-20T08:59:28.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.155+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8410cae0 con 0x7fce840f6730 2021-11-20T08:59:28.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.156+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce840f4880 con 0x7fce840f6730 2021-11-20T08:59:28.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.156+0000 7fce6b7fe700 1 -- 172.21.15.93:0/4217031 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fce78007b20 con 0x7fce840f6730 2021-11-20T08:59:28.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.157+0000 7fce6b7fe700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce6c05c300 0x7fce6c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:28.164 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.157+0000 7fce6b7fe700 1 -- 172.21.15.93:0/4217031 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fce78076140 con 0x7fce840f6730 2021-11-20T08:59:28.164 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.157+0000 7fce82ffd700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce6c05c300 0x7fce6c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:28.164 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.157+0000 7fce82ffd700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce6c05c300 0x7fce6c05e7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fce8405fe70 tx=0x7fce74009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:28.164 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.160+0000 7fce6b7fe700 1 -- 172.21.15.93:0/4217031 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce78048b00 con 0x7fce840f6730 2021-11-20T08:59:28.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.351+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce84045ad0 con 0x7fce840f6730 2021-11-20T08:59:28.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.352+0000 7fce6b7fe700 1 -- 172.21.15.93:0/4217031 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7fce7804c120 con 0x7fce840f6730 2021-11-20T08:59:28.352 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:28.352 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:28.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce6c05c300 msgr2=0x7fce6c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:28.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce6c05c300 0x7fce6c05e7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fce8405fe70 tx=0x7fce74009b30).stop 2021-11-20T08:59:28.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f6730 msgr2=0x7fce8410bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:28.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f6730 0x7fce8410bfa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fce780026d0 tx=0x7fce78009de0).stop 2021-11-20T08:59:28.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 shutdown_connections 2021-11-20T08:59:28.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fce6c05c300 0x7fce6c05e7c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:28.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce840f5490 0x7fce84109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:28.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 --2- 172.21.15.93:0/4217031 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce840f6730 0x7fce8410bfa0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:28.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 >> 172.21.15.93:0/4217031 conn(0x7fce840f0bd0 msgr2=0x7fce840fa180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:28.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 shutdown_connections 2021-11-20T08:59:28.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:28.353+0000 7fce89759700 1 -- 172.21.15.93:0/4217031 wait complete. 2021-11-20T08:59:28.357 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:28.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:28.365 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:28.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:28 smithi102 conmon[36472]: cluster 2021-11-20T08:59:27.561362+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:59:28.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:28 smithi102 conmon[36472]: 14162) 267 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T08:59:28.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:28 smithi102 conmon[36472]: audit 2021-11-20T08:59:28.353711+0000 mon.smithi093 (mon.0) 702 : audit [DBG] from='client.? 172.21.15.93:0/4217031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:28.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:28 smithi093 conmon[33522]: cluster 2021-11-20T08:59:27.561362+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:59:28.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:28 smithi093 conmon[33522]: 267 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T08:59:28.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:28 smithi093 conmon[33522]: audit 2021-11-20T08:59:28.353711+0000 mon.smithi093 (mon.0) 702 : audit [DBG] from='client.? 172.21.15.93:0/4217031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:29.366 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:29.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 -- 172.21.15.93:0/2381119277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 msgr2=0x7f1f280f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:29.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/2381119277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f280f5bd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1f18004660 tx=0x7f1f18009b30).stop 2021-11-20T08:59:29.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 -- 172.21.15.93:0/2381119277 shutdown_connections 2021-11-20T08:59:29.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/2381119277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f280f5bd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:29.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/2381119277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f280f6a50 0x7f1f280f6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:29.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 -- 172.21.15.93:0/2381119277 >> 172.21.15.93:0/2381119277 conn(0x7f1f280f0360 msgr2=0x7f1f280f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:29.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 -- 172.21.15.93:0/2381119277 shutdown_connections 2021-11-20T08:59:29.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.492+0000 7f1f2efb1700 1 -- 172.21.15.93:0/2381119277 wait complete. 2021-11-20T08:59:29.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2efb1700 1 Processor -- start 2021-11-20T08:59:29.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2efb1700 1 -- start start 2021-11-20T08:59:29.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2efb1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f28109ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:29.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2efb1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f280f6a50 0x7f1f2810bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:29.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2efb1700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280f5490 con 0x7f1f280f5750 2021-11-20T08:59:29.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2efb1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280f55d0 con 0x7f1f280f6a50 2021-11-20T08:59:29.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2dfaf700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f28109ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:29.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2dfaf700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f28109ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43802/0 (socket says 172.21.15.93:43802) 2021-11-20T08:59:29.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2dfaf700 1 -- 172.21.15.93:0/462940792 learned_addr learned my addr 172.21.15.93:0/462940792 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:29.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2d7ae700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f280f6a50 0x7f1f2810bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:29.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2dfaf700 1 -- 172.21.15.93:0/462940792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f280f6a50 msgr2=0x7f1f2810bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:29.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2dfaf700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f280f6a50 0x7f1f2810bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:29.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2dfaf700 1 -- 172.21.15.93:0/462940792 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f18005040 con 0x7f1f280f5750 2021-11-20T08:59:29.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.493+0000 7f1f2d7ae700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f280f6a50 0x7f1f2810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:59:29.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.494+0000 7f1f2dfaf700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f28109ae0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f1f240093f0 tx=0x7f1f24006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:29.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.494+0000 7f1f1e7fc700 1 -- 172.21.15.93:0/462940792 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f24009ea0 con 0x7f1f280f5750 2021-11-20T08:59:29.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.494+0000 7f1f1e7fc700 1 -- 172.21.15.93:0/462940792 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f24013af0 con 0x7f1f280f5750 2021-11-20T08:59:29.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.494+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2810c640 con 0x7f1f280f5750 2021-11-20T08:59:29.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.494+0000 7f1f1e7fc700 1 -- 172.21.15.93:0/462940792 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f2400dad0 con 0x7f1f280f5750 2021-11-20T08:59:29.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.494+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2810cbd0 con 0x7f1f280f5750 2021-11-20T08:59:29.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.495+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f280f3ff0 con 0x7f1f280f5750 2021-11-20T08:59:29.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.495+0000 7f1f1e7fc700 1 -- 172.21.15.93:0/462940792 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1f2401a060 con 0x7f1f280f5750 2021-11-20T08:59:29.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.495+0000 7f1f1e7fc700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1f1405c300 0x7f1f1405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:29.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.496+0000 7f1f1e7fc700 1 -- 172.21.15.93:0/462940792 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1f24076150 con 0x7f1f280f5750 2021-11-20T08:59:29.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.496+0000 7f1f2d7ae700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1f1405c300 0x7f1f1405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:29.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.496+0000 7f1f2d7ae700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1f1405c300 0x7f1f1405e7c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f1f18004e30 tx=0x7f1f18009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:29.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.499+0000 7f1f1e7fc700 1 -- 172.21.15.93:0/462940792 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f24048b10 con 0x7f1f280f5750 2021-11-20T08:59:29.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.686+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f28045ad0 con 0x7f1f280f5750 2021-11-20T08:59:29.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.687+0000 7f1f1e7fc700 1 -- 172.21.15.93:0/462940792 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f1f2404c130 con 0x7f1f280f5750 2021-11-20T08:59:29.687 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:29.688 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:29.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1f1405c300 msgr2=0x7f1f1405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:29.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1f1405c300 0x7f1f1405e7c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f1f18004e30 tx=0x7f1f18009b30).stop 2021-11-20T08:59:29.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 msgr2=0x7f1f28109ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:29.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f28109ae0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f1f240093f0 tx=0x7f1f24006ec0).stop 2021-11-20T08:59:29.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 shutdown_connections 2021-11-20T08:59:29.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1f1405c300 0x7f1f1405e7c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:29.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f280f5750 0x7f1f28109ae0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:29.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 --2- 172.21.15.93:0/462940792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f280f6a50 0x7f1f2810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:29.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 >> 172.21.15.93:0/462940792 conn(0x7f1f280f0360 msgr2=0x7f1f280fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:29.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 shutdown_connections 2021-11-20T08:59:29.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:29.688+0000 7f1f2efb1700 1 -- 172.21.15.93:0/462940792 wait complete. 2021-11-20T08:59:29.692 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:29.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:29.701 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:30.702 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:30.841 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 -- 172.21.15.93:0/1136543464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 msgr2=0x7febf00f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:30.841 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 --2- 172.21.15.93:0/1136543464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f6c80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7febd8004660 tx=0x7febd8009b30).stop 2021-11-20T08:59:30.841 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 -- 172.21.15.93:0/1136543464 shutdown_connections 2021-11-20T08:59:30.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 --2- 172.21.15.93:0/1136543464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f6c80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:30.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 --2- 172.21.15.93:0/1136543464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00f5580 0x7febf00f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:30.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 -- 172.21.15.93:0/1136543464 >> 172.21.15.93:0/1136543464 conn(0x7febf00f0ce0 msgr2=0x7febf00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:30.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 -- 172.21.15.93:0/1136543464 shutdown_connections 2021-11-20T08:59:30.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 -- 172.21.15.93:0/1136543464 wait complete. 2021-11-20T08:59:30.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.836+0000 7febf5ee2700 1 Processor -- start 2021-11-20T08:59:30.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febf5ee2700 1 -- start start 2021-11-20T08:59:30.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febf5ee2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00f5580 0x7febf00f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:30.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febf5ee2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:30.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febf5ee2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf00fa2d0 con 0x7febf00f6820 2021-11-20T08:59:30.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febf5ee2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf00f8870 con 0x7febf00f5580 2021-11-20T08:59:30.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febeeffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:30.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febeeffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43814/0 (socket says 172.21.15.93:43814) 2021-11-20T08:59:30.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febeeffd700 1 -- 172.21.15.93:0/3269820120 learned_addr learned my addr 172.21.15.93:0/3269820120 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:30.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febef7fe700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00f5580 0x7febf00f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:30.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febeeffd700 1 -- 172.21.15.93:0/3269820120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00f5580 msgr2=0x7febf00f9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:30.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febeeffd700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00f5580 0x7febf00f9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:30.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febeeffd700 1 -- 172.21.15.93:0/3269820120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd8005040 con 0x7febf00f6820 2021-11-20T08:59:30.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febeeffd700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f8380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7febd8009f50 tx=0x7febd8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:30.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.837+0000 7febe7fff700 1 -- 172.21.15.93:0/3269820120 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd8018070 con 0x7febf00f6820 2021-11-20T08:59:30.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.838+0000 7febe7fff700 1 -- 172.21.15.93:0/3269820120 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febd8013410 con 0x7febf00f6820 2021-11-20T08:59:30.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.838+0000 7febe7fff700 1 -- 172.21.15.93:0/3269820120 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd801c3c0 con 0x7febf00f6820 2021-11-20T08:59:30.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.838+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf00f8a90 con 0x7febf00f6820 2021-11-20T08:59:30.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.838+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf00f8ff0 con 0x7febf00f6820 2021-11-20T08:59:30.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.838+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf0046330 con 0x7febf00f6820 2021-11-20T08:59:30.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.839+0000 7febe7fff700 1 -- 172.21.15.93:0/3269820120 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7febd801d420 con 0x7febf00f6820 2021-11-20T08:59:30.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.839+0000 7febe7fff700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7febdc05c2a0 0x7febdc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:30.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.839+0000 7febe7fff700 1 -- 172.21.15.93:0/3269820120 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7febd8077e40 con 0x7febf00f6820 2021-11-20T08:59:30.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.839+0000 7febef7fe700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7febdc05c2a0 0x7febdc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:30.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.840+0000 7febef7fe700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7febdc05c2a0 0x7febdc05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7febf00f6680 tx=0x7febe0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:30.858 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:30.843+0000 7febe7fff700 1 -- 172.21.15.93:0/3269820120 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febd804a800 con 0x7febf00f6820 2021-11-20T08:59:30.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:30 smithi102 conmon[36472]: cluster 2021-11-20T08:59:29.561763+0000 mgr.smithi093.qdikkm (mgr.14162) 268 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:59:30.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:30 smithi102 conmon[36472]: audit 2021-11-20T08:59:29.688588+0000 mon.smithi093 (mon.0) 703 : audit [DBG] from='client.? 172.21.15.93:0/462940792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:30.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:30 smithi093 conmon[33522]: cluster 2021-11-20T08:59:29.561763+0000 mgr.smithi093.qdikkm (mgr.14162) 268 : cluster [DBG] 2021-11-20T08:59:30.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:30 smithi093 conmon[33522]: pgmap v240: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:59:30.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:30 smithi093 conmon[33522]: audit 2021-11-20T08:59:29.688588+0000 mon.smithi093 (mon.0) 703 : audit [DBG] from='client.? 172.21.15.93:0/462940792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:31.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.028+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febf00fdd10 con 0x7febf00f6820 2021-11-20T08:59:31.030 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.029+0000 7febe7fff700 1 -- 172.21.15.93:0/3269820120 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7febd804de20 con 0x7febf00f6820 2021-11-20T08:59:31.031 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:31.031 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:31.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7febdc05c2a0 msgr2=0x7febdc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:31.034 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7febdc05c2a0 0x7febdc05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7febf00f6680 tx=0x7febe0009040).stop 2021-11-20T08:59:31.034 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 msgr2=0x7febf00f8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:31.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f8380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7febd8009f50 tx=0x7febd8009b30).stop 2021-11-20T08:59:31.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 shutdown_connections 2021-11-20T08:59:31.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7febdc05c2a0 0x7febdc05e760 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:31.036 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf00f5580 0x7febf00f9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:31.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 --2- 172.21.15.93:0/3269820120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7febf00f6820 0x7febf00f8380 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:31.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 >> 172.21.15.93:0/3269820120 conn(0x7febf00f0ce0 msgr2=0x7febf005fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:31.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 shutdown_connections 2021-11-20T08:59:31.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:31.031+0000 7febf5ee2700 1 -- 172.21.15.93:0/3269820120 wait complete. 2021-11-20T08:59:31.039 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:31.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:31.046 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:31.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:31 smithi102 conmon[36472]: audit 2021-11-20T08:59:31.031023+0000 mon.smithi093 (mon.0) 704 : audit [DBG] from='client.? 172.21.15.93:0/3269820120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:31.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:31 smithi093 conmon[33522]: audit 2021-11-20T08:59:31.031023+0000 mon.smithi093 (mon.0 2021-11-20T08:59:31.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:31 smithi093 conmon[33522]: ) 704 : audit [DBG] from='client.? 172.21.15.93:0/3269820120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:32.047 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:32.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 -- 172.21.15.93:0/2118182259 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 msgr2=0x7f6fd40f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:32.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 --2- 172.21.15.93:0/2118182259 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd40f4bd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f6fc8004660 tx=0x7f6fc8009b30).stop 2021-11-20T08:59:32.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 -- 172.21.15.93:0/2118182259 shutdown_connections 2021-11-20T08:59:32.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 --2- 172.21.15.93:0/2118182259 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd40f4bd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:32.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 --2- 172.21.15.93:0/2118182259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd40f5a00 0x7f6fd40f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:32.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 -- 172.21.15.93:0/2118182259 >> 172.21.15.93:0/2118182259 conn(0x7f6fd40f0410 msgr2=0x7f6fd40f2890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:32.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 -- 172.21.15.93:0/2118182259 shutdown_connections 2021-11-20T08:59:32.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 -- 172.21.15.93:0/2118182259 wait complete. 2021-11-20T08:59:32.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 Processor -- start 2021-11-20T08:59:32.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.172+0000 7f6fdb386700 1 -- start start 2021-11-20T08:59:32.178 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fdb386700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd410d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:32.178 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fdb386700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd40f5a00 0x7f6fd410d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:32.178 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fdb386700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd410df10 con 0x7f6fd40f4750 2021-11-20T08:59:32.178 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fdb386700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd410e050 con 0x7f6fd40f5a00 2021-11-20T08:59:32.179 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd9122700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd410d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:32.179 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd9122700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd410d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43822/0 (socket says 172.21.15.93:43822) 2021-11-20T08:59:32.179 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd9122700 1 -- 172.21.15.93:0/1505437168 learned_addr learned my addr 172.21.15.93:0/1505437168 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:32.179 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd8921700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd40f5a00 0x7f6fd410d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:32.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd9122700 1 -- 172.21.15.93:0/1505437168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd40f5a00 msgr2=0x7f6fd410d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:32.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd9122700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd40f5a00 0x7f6fd410d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:32.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd9122700 1 -- 172.21.15.93:0/1505437168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc8005040 con 0x7f6fd40f4750 2021-11-20T08:59:32.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd9122700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd410d470 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f6fc400c530 tx=0x7f6fc4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:32.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd1ffb700 1 -- 172.21.15.93:0/1505437168 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc400f650 con 0x7f6fd40f4750 2021-11-20T08:59:32.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd1ffb700 1 -- 172.21.15.93:0/1505437168 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fc400fc60 con 0x7f6fd40f4750 2021-11-20T08:59:32.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fd1ffb700 1 -- 172.21.15.93:0/1505437168 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc4012d30 con 0x7f6fd40f4750 2021-11-20T08:59:32.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd4112af0 con 0x7f6fd40f4750 2021-11-20T08:59:32.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.173+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd4113030 con 0x7f6fd40f4750 2021-11-20T08:59:32.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.174+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd4027a00 con 0x7f6fd40f4750 2021-11-20T08:59:32.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.174+0000 7f6fd1ffb700 1 -- 172.21.15.93:0/1505437168 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6fc4010070 con 0x7f6fd40f4750 2021-11-20T08:59:32.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.175+0000 7f6fd1ffb700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6fbc05c2a0 0x7f6fbc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:32.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.175+0000 7f6fd1ffb700 1 -- 172.21.15.93:0/1505437168 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6fc4075e50 con 0x7f6fd40f4750 2021-11-20T08:59:32.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.175+0000 7f6fd8921700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6fbc05c2a0 0x7f6fbc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:32.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.175+0000 7f6fd8921700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6fbc05c2a0 0x7f6fbc05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6fc8013580 tx=0x7f6fc8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:32.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.179+0000 7f6fd1ffb700 1 -- 172.21.15.93:0/1505437168 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fc40452b0 con 0x7f6fd40f4750 2021-11-20T08:59:32.367 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.367+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fd4105470 con 0x7f6fd40f4750 2021-11-20T08:59:32.368 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.368+0000 7f6fd1ffb700 1 -- 172.21.15.93:0/1505437168 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f6fc4048890 con 0x7f6fd40f4750 2021-11-20T08:59:32.368 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:32.369 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:17.608115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14562":{"gid":14562,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6827/1373400403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":1373400403},{"type":"v1","addr":"172.21.15.93:6827","nonce":1373400403}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:32.370 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6fbc05c2a0 msgr2=0x7f6fbc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:32.370 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6fbc05c2a0 0x7f6fbc05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6fc8013580 tx=0x7f6fc8009b30).stop 2021-11-20T08:59:32.370 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 msgr2=0x7f6fd410d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:32.371 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd410d470 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f6fc400c530 tx=0x7f6fc4007e20).stop 2021-11-20T08:59:32.371 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 shutdown_connections 2021-11-20T08:59:32.371 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6fbc05c2a0 0x7f6fbc05e760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:32.371 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fd40f4750 0x7f6fd410d470 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:32.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 --2- 172.21.15.93:0/1505437168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd40f5a00 0x7f6fd410d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:32.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 >> 172.21.15.93:0/1505437168 conn(0x7f6fd40f0410 msgr2=0x7f6fd40f91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:32.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 shutdown_connections 2021-11-20T08:59:32.372 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:32.370+0000 7f6fdb386700 1 -- 172.21.15.93:0/1505437168 wait complete. 2021-11-20T08:59:32.372 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 59 2021-11-20T08:59:32.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:17.608115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14562': {'gid': 14562, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6827/1373400403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 1373400403}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 1373400403}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:32.383 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14562:up:clientreplay 2021-11-20T08:59:32.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:32 smithi102 conmon[36472]: cluster 2021-11-20T08:59:31.562374+0000 mgr.smithi093.qdikkm (mgr.14162) 269 : cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:59:32.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:32 smithi102 conmon[36472]: cluster 2021-11-20T08:59: 2021-11-20T08:59:32.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:32 smithi102 conmon[36472]: 32.265137+0000 mon.smithi093 (mon.0) 705 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:59:32.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:32 smithi102 conmon[36472]: cluster 2021-11-20T08:59:32.270954+0000 mon.smithi093 (mon.0) 706 : cluster [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-20T08:59:32.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:32 smithi102 conmon[36472]: audit 2021-11-20T08:59:32.369831+0000 2021-11-20T08:59:32.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:32 smithi102 conmon[36472]: mon.smithi093 (mon.0) 707 : audit [DBG] from='client.? 172.21.15.93:0/1505437168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:32.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: debug 2021-11-20T08:59:32.648+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: cluster 2021-11-20T08:59:31.562374 2021-11-20T08:59:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: +0000 mgr.smithi093.qdikkm (mgr.14162) 269 : cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2021-11-20T08:59:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: cluster 2021-11-20T08:59:32.265137+0000 mon.smithi093 (mon.0 2021-11-20T08:59:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: ) 705 : cluster [INF] daemon mds.cephfs.smithi093.ljzugc restarted 2021-11-20T08:59:32.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: cluster 2021- 2021-11-20T08:59:32.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: 11-20T08:59:32.270954+0000 mon.smithi093 (mon.0) 706 : cluster 2021-11-20T08:59:32.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-20T08:59:32.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: audit 2021-11-20T08:59:32.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: 2021-11-20T08:59:32. 2021-11-20T08:59:32.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: 369831+0000 mon.smithi093 (mon.0) 707 : audit [DBG] from='client.? 172.21.15.93:0/1505437168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:32.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:32 smithi093 conmon[33522]: 2021-11-20T08:59:33.384 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:33.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/3143471055 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 msgr2=0x7ff3f40f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:33.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/3143471055 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f40f7da0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff3e8004660 tx=0x7ff3e8009b30).stop 2021-11-20T08:59:33.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/3143471055 shutdown_connections 2021-11-20T08:59:33.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/3143471055 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f40f7da0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:33.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/3143471055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3f40f66a0 0x7ff3f40f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:33.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/3143471055 >> 172.21.15.93:0/3143471055 conn(0x7ff3f40f1da0 msgr2=0x7ff3f40f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:33.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/3143471055 shutdown_connections 2021-11-20T08:59:33.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/3143471055 wait complete. 2021-11-20T08:59:33.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 Processor -- start 2021-11-20T08:59:33.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.510+0000 7ff3fbdcf700 1 -- start start 2021-11-20T08:59:33.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3fbdcf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3f40f66a0 0x7ff3f4106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:33.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3fbdcf700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f41073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:33.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3fbdcf700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4107960 con 0x7ff3f40f7940 2021-11-20T08:59:33.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3fbdcf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4107aa0 con 0x7ff3f40f66a0 2021-11-20T08:59:33.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f936a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f41073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:33.516 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f936a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f41073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43846/0 (socket says 172.21.15.93:43846) 2021-11-20T08:59:33.516 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f936a700 1 -- 172.21.15.93:0/647712700 learned_addr learned my addr 172.21.15.93:0/647712700 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:33.516 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f936a700 1 -- 172.21.15.93:0/647712700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3f40f66a0 msgr2=0x7ff3f4106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:33.516 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f936a700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3f40f66a0 0x7ff3f4106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:33.517 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f936a700 1 -- 172.21.15.93:0/647712700 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e8005040 con 0x7ff3f40f7940 2021-11-20T08:59:33.517 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f936a700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f41073b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7ff3e8009830 tx=0x7ff3e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:33.517 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f27fc700 1 -- 172.21.15.93:0/647712700 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e8018070 con 0x7ff3f40f7940 2021-11-20T08:59:33.517 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f27fc700 1 -- 172.21.15.93:0/647712700 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3e8013410 con 0x7ff3f40f7940 2021-11-20T08:59:33.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f410c4e0 con 0x7ff3f40f7940 2021-11-20T08:59:33.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3f27fc700 1 -- 172.21.15.93:0/647712700 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e801c620 con 0x7ff3f40f7940 2021-11-20T08:59:33.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.511+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f410caa0 con 0x7ff3f40f7940 2021-11-20T08:59:33.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.512+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f4027a00 con 0x7ff3f40f7940 2021-11-20T08:59:33.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.512+0000 7ff3f27fc700 1 -- 172.21.15.93:0/647712700 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff3e801c8e0 con 0x7ff3f40f7940 2021-11-20T08:59:33.519 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.513+0000 7ff3f27fc700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3dc05c2a0 0x7ff3dc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:33.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.513+0000 7ff3f27fc700 1 -- 172.21.15.93:0/647712700 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7ff3e8078100 con 0x7ff3f40f7940 2021-11-20T08:59:33.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.513+0000 7ff3f9b6b700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3dc05c2a0 0x7ff3dc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:33.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.513+0000 7ff3f9b6b700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3dc05c2a0 0x7ff3dc05e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff3f40f77a0 tx=0x7ff3e40049e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:33.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.517+0000 7ff3f27fc700 1 -- 172.21.15.93:0/647712700 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3e804aa00 con 0x7ff3f40f7940 2021-11-20T08:59:33.662 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: cluster 2021-11-20T08:59:32.649951+0000 mon.smithi093 2021-11-20T08:59:33.662 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: (mon.0) 708 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:33.663 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: cluster 2021-11-20T08:59:32.655247+0000 mon.smithi093 (mon.0) 709 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:boot 2021-11-20T08:59:33.663 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: cluster 2021-11-20T08:59:33.664 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: 2021-11-20T08:59:32.655305+0000 mon.smithi093 (mon.0) 710 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:59:33.664 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T08:59:33.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: 08:59:32.655401+0000 mon.smithi093 (mon.0 2021-11-20T08:59:33.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: ) 711 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:59:33.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: cluster 2021-11-20T08 2021-11-20T08:59:33.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: :59:32.655645+0000 mon.smithi093 (mon.0 2021-11-20T08:59:33.665 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: ) 712 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:59:33.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: audit 2021-11-20T08: 2021-11-20T08:59:33.666 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: 59:32.655756+0000 mon.smithi093 (mon.0) 2021-11-20T08:59:33.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.716+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3f40fee30 con 0x7ff3f40f7940 2021-11-20T08:59:33.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.716+0000 7ff3f27fc700 1 -- 172.21.15.93:0/647712700 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2939 (secure 0 0 0) 0x7ff3e804e020 con 0x7ff3f40f7940 2021-11-20T08:59:33.718 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:33.718 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":60},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:33.658662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:33.720 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3dc05c2a0 msgr2=0x7ff3dc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:33.720 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3dc05c2a0 0x7ff3dc05e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff3f40f77a0 tx=0x7ff3e40049e0).stop 2021-11-20T08:59:33.720 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 msgr2=0x7ff3f41073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:33.720 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f41073b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7ff3e8009830 tx=0x7ff3e8009b30).stop 2021-11-20T08:59:33.721 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 shutdown_connections 2021-11-20T08:59:33.721 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3dc05c2a0 0x7ff3dc05e760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:33.721 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3f40f66a0 0x7ff3f4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:33.721 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 --2- 172.21.15.93:0/647712700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f40f7940 0x7ff3f41073b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:33.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 >> 172.21.15.93:0/647712700 conn(0x7ff3f40f1da0 msgr2=0x7ff3f40fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:33.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 shutdown_connections 2021-11-20T08:59:33.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:33.718+0000 7ff3fbdcf700 1 -- 172.21.15.93:0/647712700 wait complete. 2021-11-20T08:59:33.723 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 62 2021-11-20T08:59:33.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:33.658662+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:33.731 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:reconnect 2021-11-20T08:59:33.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: cluster 2021-11-20T08:59:32.649951+0000 mon.smithi093 (mon.0) 708 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T08:59:33.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: cluster 2021-11-20T08:59:32.655247+0000 mon.smithi093 (mon.0) 709 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:boot 2021-11-20T08:59:33.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: cluster 2021-11-20T08:59:32.655305+0000 mon.smithi093 (mon.0 2021-11-20T08:59:33.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: ) 710 : cluster [INF] Standby daemon mds.cephfs.smithi093.zrryyo assigned to filesystem cephfs as rank 0 2021-11-20T08:59:33.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:59:33.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: -11-20T08:59:32.655401+0000 mon.smithi093 (mon 2021-11-20T08:59:33.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: .0) 711 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T08:59:33.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: cluster 2021 2021-11-20T08:59:33.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: -11-20T08:59: 2021-11-20T08:59:33.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: 32.655645+0000 mon.smithi093 (mon.0) 712 : cluster 2021-11-20T08:59:33.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T08:59:33.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T08:59:33.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: T08:59:32.655756+0000 mon.smithi093 ( 2021-11-20T08:59:33.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: mon.0) 713 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:59:33.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: cluster 2021-11-20T08:59:33.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: 2021-11-20T08:59 2021-11-20T08:59:33.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: :32.661164+0000 mon.smithi093 (mon.0) 714 2021-11-20T08:59:33.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:33 smithi102 conmon[36472]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:59:33.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: 713 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi093.ljzugc"}]: dispatch 2021-11-20T08:59:33.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: cluster 2021-11-20T08:59 2021-11-20T08:59:33.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: :32.661164+0000 mon.smithi093 (mon.0) 714 : 2021-11-20T08:59:33.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:33 smithi093 conmon[33522]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:replay} 3 up:standby 2021-11-20T08:59:34.731 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:34.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 -- 172.21.15.93:0/333024327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80600e0 msgr2=0x7f3bc80f92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:34.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 --2- 172.21.15.93:0/333024327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80600e0 0x7f3bc80f92e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f3bb80099e0 tx=0x7f3bb8004d10).stop 2021-11-20T08:59:34.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 -- 172.21.15.93:0/333024327 shutdown_connections 2021-11-20T08:59:34.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 --2- 172.21.15.93:0/333024327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc80f9860 0x7f3bc80fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:34.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 --2- 172.21.15.93:0/333024327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80600e0 0x7f3bc80f92e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:34.867 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 -- 172.21.15.93:0/333024327 >> 172.21.15.93:0/333024327 conn(0x7f3bc80f0ce0 msgr2=0x7f3bc80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:34.867 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 -- 172.21.15.93:0/333024327 shutdown_connections 2021-11-20T08:59:34.868 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 -- 172.21.15.93:0/333024327 wait complete. 2021-11-20T08:59:34.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.859+0000 7f3bcde13700 1 Processor -- start 2021-11-20T08:59:34.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bcde13700 1 -- start start 2021-11-20T08:59:34.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bcde13700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc80600e0 0x7f3bc8106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:34.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bcde13700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80f9860 0x7f3bc8107490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:34.871 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bcde13700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc8107a40 con 0x7f3bc80f9860 2021-11-20T08:59:34.871 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bcde13700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc8107b80 con 0x7f3bc80600e0 2021-11-20T08:59:34.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc6ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80f9860 0x7f3bc8107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:34.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc6ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80f9860 0x7f3bc8107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43858/0 (socket says 172.21.15.93:43858) 2021-11-20T08:59:34.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc6ffd700 1 -- 172.21.15.93:0/217263507 learned_addr learned my addr 172.21.15.93:0/217263507 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:34.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc77fe700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc80600e0 0x7f3bc8106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:34.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc6ffd700 1 -- 172.21.15.93:0/217263507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc80600e0 msgr2=0x7f3bc8106fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:34.875 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc6ffd700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc80600e0 0x7f3bc8106fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:34.875 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc6ffd700 1 -- 172.21.15.93:0/217263507 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb8009640 con 0x7f3bc80f9860 2021-11-20T08:59:34.879 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bc6ffd700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80f9860 0x7f3bc8107490 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3bb000c510 tx=0x7f3bb0007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:34.879 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bbffff700 1 -- 172.21.15.93:0/217263507 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bb000f650 con 0x7f3bc80f9860 2021-11-20T08:59:34.879 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bbffff700 1 -- 172.21.15.93:0/217263507 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bb000fc60 con 0x7f3bc80f9860 2021-11-20T08:59:34.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc810c620 con 0x7f3bc80f9860 2021-11-20T08:59:34.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.860+0000 7f3bbffff700 1 -- 172.21.15.93:0/217263507 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bb0012c00 con 0x7f3bc80f9860 2021-11-20T08:59:34.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.861+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc810cb60 con 0x7f3bc80f9860 2021-11-20T08:59:34.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.861+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc8027a00 con 0x7f3bc80f9860 2021-11-20T08:59:34.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.862+0000 7f3bbffff700 1 -- 172.21.15.93:0/217263507 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3bb0010070 con 0x7f3bc80f9860 2021-11-20T08:59:34.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.862+0000 7f3bbffff700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3bb405c2a0 0x7f3bb405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:34.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.862+0000 7f3bbffff700 1 -- 172.21.15.93:0/217263507 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f3bb0075ed0 con 0x7f3bc80f9860 2021-11-20T08:59:34.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.862+0000 7f3bc77fe700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3bb405c2a0 0x7f3bb405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:34.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.863+0000 7f3bc77fe700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3bb405c2a0 0x7f3bb405e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f3bb8003380 tx=0x7f3bb8004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:34.882 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:34.867+0000 7f3bbffff700 1 -- 172.21.15.93:0/217263507 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bb00452b0 con 0x7f3bc80f9860 2021-11-20T08:59:34.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: cluster 2021-11-20T08:59:33.562735+0000 mgr.smithi093.qdikkm (mgr.14162) 270 : 2021-11-20T08:59:34.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: cluster [DBG] pgmap v243: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T08:59:34.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: cluster 2021-11-20T08:59:33.665017+0000 mon.smithi093 (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:reconnect 2021-11-20T08:59:34.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: cluster 2021-11-20T08:59:34.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: 2021-11-20T08:59:33.665071+0000 mon.smithi093 (mon.0) 716 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:59:34.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: cluster 2021-11-20T08:59:33.684435+0000 mds.cephfs.smithi093.zrryyo 2021-11-20T08:59:34.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:59:34.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: audit 2021 2021-11-20T08:59:34.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: -11-20T08:59:33.718226+0000 2021-11-20T08:59:34.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: mon.smithi093 (mon.0) 717 : audit [DBG] from='client.? 172.21.15.93:0/647712700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:34.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:34 smithi102 conmon[36472]: 2021-11-20T08:59:34.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: cluster 2021-11-20T08:59:33.562735+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:59:34.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: ) 270 : cluster [DBG] pgmap v243: 65 pgs: 65 active+clean; 2.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T08:59:34.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: cluster 2021-11-20T08:59:33.665017+0000 mon.smithi093 (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:reconnect 2021-11-20T08:59:34.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: cluster 2021-11-20T08:59:33.665071+0000 mon.smithi093 (mon.0) 716 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:reconnect} 3 up:standby 2021-11-20T08:59:34.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: cluster 2021-11-20T08:59:34.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: 2021-11-20T08:59:33.684435+0000 mds.cephfs.smithi093.zrryyo (mds.0 2021-11-20T08:59:34.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: ) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T08:59:34.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: audit 2021-11-20T08:59:34.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: 2021-11-20T08:59:33.718226+0000 mon.smithi093 2021-11-20T08:59:34.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:34 smithi093 conmon[33522]: (mon.0) 717 : audit [DBG] from='client.? 172.21.15.93:0/647712700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:35.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.049+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bc80fee50 con 0x7f3bc80f9860 2021-11-20T08:59:35.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.050+0000 7f3bbffff700 1 -- 172.21.15.93:0/217263507 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2936 (secure 0 0 0) 0x7f3bb004c260 con 0x7f3bc80f9860 2021-11-20T08:59:35.050 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:35.050 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":60},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:34.662923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:35.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3bb405c2a0 msgr2=0x7f3bb405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:35.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3bb405c2a0 0x7f3bb405e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f3bb8003380 tx=0x7f3bb8004d10).stop 2021-11-20T08:59:35.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80f9860 msgr2=0x7f3bc8107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:35.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80f9860 0x7f3bc8107490 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3bb000c510 tx=0x7f3bb0007df0).stop 2021-11-20T08:59:35.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 shutdown_connections 2021-11-20T08:59:35.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3bb405c2a0 0x7f3bb405e760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:35.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc80600e0 0x7f3bc8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:35.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 --2- 172.21.15.93:0/217263507 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bc80f9860 0x7f3bc8107490 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:35.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 >> 172.21.15.93:0/217263507 conn(0x7f3bc80f0ce0 msgr2=0x7f3bc80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:35.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 shutdown_connections 2021-11-20T08:59:35.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:35.051+0000 7f3bcde13700 1 -- 172.21.15.93:0/217263507 wait complete. 2021-11-20T08:59:35.055 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 63 2021-11-20T08:59:35.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:34.662923+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:35.063 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:rejoin 2021-11-20T08:59:35.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:35 smithi093 conmon[33522]: cluster 2021-11-20T08:59:34.666536+0000 mon.smithi093 (mon.0) 718 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:rejoin 2021-11-20T08:59:35.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:35 smithi093 conmon[33522]: cluster 2021-11-20T08:59:34.666601+0000 mon.smithi093 (mon.0) 719 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:59:35.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:35 smithi093 conmon[33522]: audit 2021-11-20T08:59:35.051528+0000 mon.smithi093 (mon.0) 720 : audit [DBG] from='client.? 172.21.15.93:0/217263507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:36.065 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:36.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:35 smithi102 conmon[36472]: cluster 2021-11-20T08:59:34.666536+0000 mon.smithi093 (mon.0) 718 : cluster [DBG] 2021-11-20T08:59:36.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:35 smithi102 conmon[36472]: mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:rejoin 2021-11-20T08:59:36.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:35 smithi102 conmon[36472]: cluster 2021-11-20T08:59:34.666601+0000 mon.smithi093 (mon.0) 719 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:rejoin} 3 up:standby 2021-11-20T08:59:36.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:35 smithi102 conmon[36472]: audit 2021-11-20T08:59:35 2021-11-20T08:59:36.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:35 smithi102 conmon[36472]: .051528+0000 mon.smithi093 (mon.0) 720 : audit [DBG] from='client.? 172.21.15.93:0/217263507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:36.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 -- 172.21.15.93:0/2764731096 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 msgr2=0x7f76380f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:36.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 --2- 172.21.15.93:0/2764731096 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f76380f4d80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f7628004660 tx=0x7f7628009b30).stop 2021-11-20T08:59:36.189 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 -- 172.21.15.93:0/2764731096 shutdown_connections 2021-11-20T08:59:36.189 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 --2- 172.21.15.93:0/2764731096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76380f5300 0x7f76380f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:36.190 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 --2- 172.21.15.93:0/2764731096 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f76380f4d80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:36.190 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 -- 172.21.15.93:0/2764731096 >> 172.21.15.93:0/2764731096 conn(0x7f76380f0440 msgr2=0x7f76380f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:36.191 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 -- 172.21.15.93:0/2764731096 shutdown_connections 2021-11-20T08:59:36.191 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.182+0000 7f763da41700 1 -- 172.21.15.93:0/2764731096 wait complete. 2021-11-20T08:59:36.191 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f763da41700 1 Processor -- start 2021-11-20T08:59:36.192 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f763da41700 1 -- start start 2021-11-20T08:59:36.192 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f763da41700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f7638106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:36.193 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f763da41700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76380f5300 0x7f76381073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:36.193 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f763da41700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638107990 con 0x7f76380f4960 2021-11-20T08:59:36.194 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f763da41700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638107ad0 con 0x7f76380f5300 2021-11-20T08:59:36.194 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f7636ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f7638106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:36.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f7636ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f7638106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43866/0 (socket says 172.21.15.93:43866) 2021-11-20T08:59:36.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f7636ffd700 1 -- 172.21.15.93:0/2443630969 learned_addr learned my addr 172.21.15.93:0/2443630969 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:36.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.183+0000 7f76367fc700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76380f5300 0x7f76381073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:36.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f7636ffd700 1 -- 172.21.15.93:0/2443630969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76380f5300 msgr2=0x7f76381073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:36.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f7636ffd700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76380f5300 0x7f76381073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:36.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f7636ffd700 1 -- 172.21.15.93:0/2443630969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628005040 con 0x7f76380f4960 2021-11-20T08:59:36.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f7636ffd700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f7638106ef0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f7628004e30 tx=0x7f7628009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:36.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f76277fe700 1 -- 172.21.15.93:0/2443630969 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7628018070 con 0x7f76380f4960 2021-11-20T08:59:36.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f76277fe700 1 -- 172.21.15.93:0/2443630969 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7628013410 con 0x7f76380f4960 2021-11-20T08:59:36.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763810c510 con 0x7f76380f4960 2021-11-20T08:59:36.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f76277fe700 1 -- 172.21.15.93:0/2443630969 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f762801de20 con 0x7f76380f4960 2021-11-20T08:59:36.200 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.184+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763810cad0 con 0x7f76380f4960 2021-11-20T08:59:36.200 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.185+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7638027a00 con 0x7f76380f4960 2021-11-20T08:59:36.200 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.185+0000 7f76277fe700 1 -- 172.21.15.93:0/2443630969 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f762801d420 con 0x7f76380f4960 2021-11-20T08:59:36.201 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.185+0000 7f76277fe700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f761c05c250 0x7f761c05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:36.201 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.185+0000 7f76277fe700 1 -- 172.21.15.93:0/2443630969 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f7628077ea0 con 0x7f76380f4960 2021-11-20T08:59:36.202 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.185+0000 7f76367fc700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f761c05c250 0x7f761c05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:36.202 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.186+0000 7f76367fc700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f761c05c250 0x7f761c05e710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f76380f5740 tx=0x7f762c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:36.203 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.189+0000 7f76277fe700 1 -- 172.21.15.93:0/2443630969 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f762804a800 con 0x7f76380f4960 2021-11-20T08:59:36.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.379+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76380fee70 con 0x7f76380f4960 2021-11-20T08:59:36.378 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.379+0000 7f76277fe700 1 -- 172.21.15.93:0/2443630969 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2942 (secure 0 0 0) 0x7f762804de20 con 0x7f76380f4960 2021-11-20T08:59:36.379 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:36.379 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":60},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:36.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f761c05c250 msgr2=0x7f761c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:36.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f761c05c250 0x7f761c05e710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f76380f5740 tx=0x7f762c00b040).stop 2021-11-20T08:59:36.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 msgr2=0x7f7638106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:36.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f7638106ef0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f7628004e30 tx=0x7f7628009b30).stop 2021-11-20T08:59:36.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 shutdown_connections 2021-11-20T08:59:36.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f761c05c250 0x7f761c05e710 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:36.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76380f4960 0x7f7638106ef0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:36.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 --2- 172.21.15.93:0/2443630969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76380f5300 0x7f76381073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:36.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 >> 172.21.15.93:0/2443630969 conn(0x7f76380f0440 msgr2=0x7f76380fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:36.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 shutdown_connections 2021-11-20T08:59:36.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:36.381+0000 7f763da41700 1 -- 172.21.15.93:0/2443630969 wait complete. 2021-11-20T08:59:36.383 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 64 2021-11-20T08:59:36.393 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:36.393 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:36.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: cluster 2021-11-20T08:59:35.563375+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T08:59:36.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: .14162) 271 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 2.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:59:36.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: cluster 2021-11-20T08:59:35.669195+0000 mon.smithi093 (mon.0) 721 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:clientreplay 2021-11-20T08:59:36.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: cluster 2021-11-20T08:59:35.669258+0000 2021-11-20T08:59:36.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: mon.smithi093 (mon.0) 722 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:59:36.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: 2021-11-20T08:59:36.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:59:36.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: 11-20T08:59:36.380198+0000 mon.smithi093 (mon.0 2021-11-20T08:59:36.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:36 smithi093 conmon[33522]: ) 723 : audit [DBG] from='client.? 172.21.15.93:0/2443630969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:37.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:36 smithi102 conmon[36472]: cluster 2021-11-20T08:59:35.563375+0000 mgr.smithi093.qdikkm (mgr.14162) 271 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 2.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:59:37.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:36 smithi102 conmon[36472]: 2021-11-20T08:59:37.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:36 smithi102 conmon[36472]: cluster 2021-11-20T08:59:35.669195+0000 mon.smithi093 (mon.0) 721 : cluster [DBG] mds.? [v2:172.21.15.93:6828/2300988268,v1:172.21.15.93:6829/2300988268] up:clientreplay 2021-11-20T08:59:37.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:36 smithi102 conmon[36472]: cluster 2021-11-20T08:59:35.669258+0000 mon.smithi093 (mon.0) 722 : 2021-11-20T08:59:37.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:36 smithi102 conmon[36472]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:59:37.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:36 smithi102 conmon[36472]: audit 2021-11-20T08:59:36. 2021-11-20T08:59:37.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:36 smithi102 conmon[36472]: 380198+0000 mon.smithi093 (mon.0) 723 : audit [DBG] from='client.? 172.21.15.93:0/2443630969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:37.394 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:37.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- 172.21.15.93:0/3573392492 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 msgr2=0x7f56e80f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:37.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/3573392492 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e80f6ca0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f56dc004660 tx=0x7f56dc009b30).stop 2021-11-20T08:59:37.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- 172.21.15.93:0/3573392492 shutdown_connections 2021-11-20T08:59:37.519 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/3573392492 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e80f6ca0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:37.519 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/3573392492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56e80f55a0 0x7f56e80f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:37.519 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- 172.21.15.93:0/3573392492 >> 172.21.15.93:0/3573392492 conn(0x7f56e80f0ce0 msgr2=0x7f56e80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:37.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- 172.21.15.93:0/3573392492 shutdown_connections 2021-11-20T08:59:37.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- 172.21.15.93:0/3573392492 wait complete. 2021-11-20T08:59:37.521 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 Processor -- start 2021-11-20T08:59:37.521 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- start start 2021-11-20T08:59:37.522 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56e80f55a0 0x7f56e8105d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:37.522 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e8106220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:37.522 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e81067d0 con 0x7f56e80f6840 2021-11-20T08:59:37.522 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.513+0000 7f56ecf2a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e8106910 con 0x7f56e80f55a0 2021-11-20T08:59:37.523 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e5d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e8106220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:37.523 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e5d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e8106220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43884/0 (socket says 172.21.15.93:43884) 2021-11-20T08:59:37.523 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e5d9b700 1 -- 172.21.15.93:0/2492623885 learned_addr learned my addr 172.21.15.93:0/2492623885 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:37.523 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e659c700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56e80f55a0 0x7f56e8105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:37.523 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e5d9b700 1 -- 172.21.15.93:0/2492623885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56e80f55a0 msgr2=0x7f56e8105d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:37.524 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e5d9b700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56e80f55a0 0x7f56e8105d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:37.524 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e5d9b700 1 -- 172.21.15.93:0/2492623885 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc005040 con 0x7f56e80f6840 2021-11-20T08:59:37.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56e5d9b700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e8106220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f56dc009f50 tx=0x7f56dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:37.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56d6ffd700 1 -- 172.21.15.93:0/2492623885 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56dc018070 con 0x7f56e80f6840 2021-11-20T08:59:37.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56d6ffd700 1 -- 172.21.15.93:0/2492623885 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56dc013410 con 0x7f56e80f6840 2021-11-20T08:59:37.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56d6ffd700 1 -- 172.21.15.93:0/2492623885 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56dc01c3c0 con 0x7f56e80f6840 2021-11-20T08:59:37.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e810b350 con 0x7f56e80f6840 2021-11-20T08:59:37.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.514+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e810b910 con 0x7f56e80f6840 2021-11-20T08:59:37.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.515+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e8046330 con 0x7f56e80f6840 2021-11-20T08:59:37.527 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.515+0000 7f56d6ffd700 1 -- 172.21.15.93:0/2492623885 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f56dc01d420 con 0x7f56e80f6840 2021-11-20T08:59:37.527 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.516+0000 7f56d6ffd700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f56d005c2a0 0x7f56d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:37.528 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.516+0000 7f56d6ffd700 1 -- 172.21.15.93:0/2492623885 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f56dc077dd0 con 0x7f56e80f6840 2021-11-20T08:59:37.528 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.516+0000 7f56e659c700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f56d005c2a0 0x7f56d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:37.528 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.516+0000 7f56e659c700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f56d005c2a0 0x7f56d005e760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f56e80f66a0 tx=0x7f56d800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:37.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.520+0000 7f56d6ffd700 1 -- 172.21.15.93:0/2492623885 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56dc04a7b0 con 0x7f56e80f6840 2021-11-20T08:59:37.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.707+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56e8027a00 con 0x7f56e80f6840 2021-11-20T08:59:37.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.707+0000 7f56d6ffd700 1 -- 172.21.15.93:0/2492623885 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f56e8027a00 con 0x7f56e80f6840 2021-11-20T08:59:37.707 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:37.708 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:37.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f56d005c2a0 msgr2=0x7f56d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:37.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f56d005c2a0 0x7f56d005e760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f56e80f66a0 tx=0x7f56d800b040).stop 2021-11-20T08:59:37.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 msgr2=0x7f56e8106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:37.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e8106220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f56dc009f50 tx=0x7f56dc009b30).stop 2021-11-20T08:59:37.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 shutdown_connections 2021-11-20T08:59:37.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f56d005c2a0 0x7f56d005e760 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:37.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56e80f55a0 0x7f56e8105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:37.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 --2- 172.21.15.93:0/2492623885 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56e80f6840 0x7f56e8106220 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:37.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 >> 172.21.15.93:0/2492623885 conn(0x7f56e80f0ce0 msgr2=0x7f56e80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:37.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 shutdown_connections 2021-11-20T08:59:37.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:37.709+0000 7f56ecf2a700 1 -- 172.21.15.93:0/2492623885 wait complete. 2021-11-20T08:59:37.712 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:37.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:37.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:37.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:37 smithi093 conmon[33522]: cluster 2021-11-20T08:59:36.678580+0000 mon.smithi093 (mon 2021-11-20T08:59:37.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:37 smithi093 conmon[33522]: .0) 724 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:standby 2021-11-20T08:59:37.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:37 smithi093 conmon[33522]: cluster 2021-11-20T08:59:36.678632+0000 mon.smithi093 (mon.0) 725 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:59:38.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:37 smithi102 conmon[36472]: cluster 2021-11-20T08:59:36.678580+0000 mon.smithi093 (mon.0) 724 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:standby 2021-11-20T08:59:38.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:37 smithi102 conmon[36472]: cluster 2021-11-20T08:59:36.678632+0000 mon.smithi093 (mon.0) 725 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.zrryyo=up:clientreplay} 3 up:standby 2021-11-20T08:59:38.722 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:38.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 -- 172.21.15.93:0/83833021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 msgr2=0x7f0d780f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:38.845 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 --2- 172.21.15.93:0/83833021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d780f59c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f0d68004660 tx=0x7f0d68009b30).stop 2021-11-20T08:59:38.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 -- 172.21.15.93:0/83833021 shutdown_connections 2021-11-20T08:59:38.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 --2- 172.21.15.93:0/83833021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d780f6840 0x7f0d780f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:38.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 --2- 172.21.15.93:0/83833021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d780f59c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:38.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 -- 172.21.15.93:0/83833021 >> 172.21.15.93:0/83833021 conn(0x7f0d780f0ce0 msgr2=0x7f0d780f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:38.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 -- 172.21.15.93:0/83833021 shutdown_connections 2021-11-20T08:59:38.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 -- 172.21.15.93:0/83833021 wait complete. 2021-11-20T08:59:38.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 Processor -- start 2021-11-20T08:59:38.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 -- start start 2021-11-20T08:59:38.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.841+0000 7f0d7e610700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d78106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:38.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d7e610700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d780f6840 0x7f0d781073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:38.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d7e610700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78107950 con 0x7f0d780f55a0 2021-11-20T08:59:38.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d7e610700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78107a90 con 0x7f0d780f6840 2021-11-20T08:59:38.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d77fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d78106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:38.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d77fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d78106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43894/0 (socket says 172.21.15.93:43894) 2021-11-20T08:59:38.852 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d77fff700 1 -- 172.21.15.93:0/568682619 learned_addr learned my addr 172.21.15.93:0/568682619 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:38.852 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d777fe700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d780f6840 0x7f0d781073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:38.853 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d77fff700 1 -- 172.21.15.93:0/568682619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d780f6840 msgr2=0x7f0d781073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:38.853 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d77fff700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d780f6840 0x7f0d781073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:38.854 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d77fff700 1 -- 172.21.15.93:0/568682619 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d68005040 con 0x7f0d780f55a0 2021-11-20T08:59:38.854 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d77fff700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d78106eb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f0d68009f50 tx=0x7f0d68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:38.855 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d74ff9700 1 -- 172.21.15.93:0/568682619 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d68018070 con 0x7f0d780f55a0 2021-11-20T08:59:38.855 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d74ff9700 1 -- 172.21.15.93:0/568682619 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d68013410 con 0x7f0d780f55a0 2021-11-20T08:59:38.855 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7810c4d0 con 0x7f0d780f55a0 2021-11-20T08:59:38.856 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d74ff9700 1 -- 172.21.15.93:0/568682619 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d6801c3c0 con 0x7f0d780f55a0 2021-11-20T08:59:38.856 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.842+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7810ca90 con 0x7f0d780f55a0 2021-11-20T08:59:38.857 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.843+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d780fdd30 con 0x7f0d780f55a0 2021-11-20T08:59:38.857 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.843+0000 7f0d74ff9700 1 -- 172.21.15.93:0/568682619 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0d6801d420 con 0x7f0d780f55a0 2021-11-20T08:59:38.858 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.844+0000 7f0d74ff9700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d6005c2a0 0x7f0d6005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:38.858 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.844+0000 7f0d74ff9700 1 -- 172.21.15.93:0/568682619 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f0d68077f90 con 0x7f0d780f55a0 2021-11-20T08:59:38.858 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.844+0000 7f0d777fe700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d6005c2a0 0x7f0d6005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:38.859 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.844+0000 7f0d777fe700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d6005c2a0 0x7f0d6005e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f0d6c007fd0 tx=0x7f0d6c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:38.860 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:38.847+0000 7f0d74ff9700 1 -- 172.21.15.93:0/568682619 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d6804a8f0 con 0x7f0d780f55a0 2021-11-20T08:59:38.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:38 smithi093 conmon[33522]: cluster 2021-11-20T08:59:37.563831+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T08:59:38.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:38 smithi093 conmon[33522]: 14162) 272 : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:38.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:38 smithi093 conmon[33522]: audit 2021-11-20T08:59:37.708230+0000 mon.smithi093 2021-11-20T08:59:38.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:38 smithi093 conmon[33522]: (mon.0) 726 : audit [DBG] from='client.? 172.21.15.93:0/2492623885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:39.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.031+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d7810d380 con 0x7f0d780f55a0 2021-11-20T08:59:39.030 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.031+0000 7f0d74ff9700 1 -- 172.21.15.93:0/568682619 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f0d6804df10 con 0x7f0d780f55a0 2021-11-20T08:59:39.031 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:39.031 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:39.032 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d6005c2a0 msgr2=0x7f0d6005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:39.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d6005c2a0 0x7f0d6005e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f0d6c007fd0 tx=0x7f0d6c009040).stop 2021-11-20T08:59:39.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 msgr2=0x7f0d78106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:39.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d78106eb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f0d68009f50 tx=0x7f0d68009b30).stop 2021-11-20T08:59:39.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 shutdown_connections 2021-11-20T08:59:39.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d6005c2a0 0x7f0d6005e760 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:39.034 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d780f55a0 0x7f0d78106eb0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:39.034 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 --2- 172.21.15.93:0/568682619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d780f6840 0x7f0d781073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:39.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 >> 172.21.15.93:0/568682619 conn(0x7f0d780f0ce0 msgr2=0x7f0d7805f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:39.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 shutdown_connections 2021-11-20T08:59:39.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:39.033+0000 7f0d7e610700 1 -- 172.21.15.93:0/568682619 wait complete. 2021-11-20T08:59:39.036 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:39.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:39.048 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:39.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:38 smithi102 conmon[36472]: cluster 2021-11-20T08:59:37.563831+0000 mgr.smithi093.qdikkm (mgr.14162) 272 : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:39.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:38 smithi102 conmon[36472]: audit 2021-11-20T08:59:37.708230+0000 mon.smithi093 (mon.0) 726 : audit [DBG] from='client.? 172.21.15.93:0/2492623885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:39.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:39 smithi093 conmon[33522]: audit 2021-11-20T08:59:39.032394+0000 mon.smithi093 (mon 2021-11-20T08:59:39.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:39 smithi093 conmon[33522]: .0) 727 : audit [DBG] from='client.? 172.21.15.93:0/568682619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:40.049 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:40.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:39 smithi102 conmon[36472]: audit 2021-11-20T08:59:39.032394+0000 mon.smithi093 (mon.0) 727 : audit [DBG] from='client.? 172.21.15.93:0/568682619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:40.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 -- 172.21.15.93:0/2167423760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 msgr2=0x7f40f40f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:40.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 --2- 172.21.15.93:0/2167423760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f40f81b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f40e8004660 tx=0x7f40e8009b30).stop 2021-11-20T08:59:40.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 -- 172.21.15.93:0/2167423760 shutdown_connections 2021-11-20T08:59:40.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 --2- 172.21.15.93:0/2167423760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40f40f4780 0x7f40f40f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:40.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 --2- 172.21.15.93:0/2167423760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f40f81b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:40.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 -- 172.21.15.93:0/2167423760 >> 172.21.15.93:0/2167423760 conn(0x7f40f40f04a0 msgr2=0x7f40f40f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:40.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 -- 172.21.15.93:0/2167423760 shutdown_connections 2021-11-20T08:59:40.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.179+0000 7f40faec4700 1 -- 172.21.15.93:0/2167423760 wait complete. 2021-11-20T08:59:40.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40faec4700 1 Processor -- start 2021-11-20T08:59:40.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40faec4700 1 -- start start 2021-11-20T08:59:40.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40faec4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40f40f4780 0x7f40f4106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:40.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40faec4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f41074b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:40.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40faec4700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4107a60 con 0x7f40f40f7d90 2021-11-20T08:59:40.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40faec4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4107ba0 con 0x7f40f40f4780 2021-11-20T08:59:40.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40f3fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f41074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:40.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40f3fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f41074b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43904/0 (socket says 172.21.15.93:43904) 2021-11-20T08:59:40.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40f3fff700 1 -- 172.21.15.93:0/517875149 learned_addr learned my addr 172.21.15.93:0/517875149 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:40.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.180+0000 7f40f8c60700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40f40f4780 0x7f40f4106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:40.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40f3fff700 1 -- 172.21.15.93:0/517875149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40f40f4780 msgr2=0x7f40f4106fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:40.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40f3fff700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40f40f4780 0x7f40f4106fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:40.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40f3fff700 1 -- 172.21.15.93:0/517875149 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e8005040 con 0x7f40f40f7d90 2021-11-20T08:59:40.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40f3fff700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f41074b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f40e0002700 tx=0x7f40e0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:40.186 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40f17fa700 1 -- 172.21.15.93:0/517875149 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e00079c0 con 0x7f40f40f7d90 2021-11-20T08:59:40.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40f17fa700 1 -- 172.21.15.93:0/517875149 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40e000f040 con 0x7f40f40f7d90 2021-11-20T08:59:40.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f410c640 con 0x7f40f40f7d90 2021-11-20T08:59:40.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40f17fa700 1 -- 172.21.15.93:0/517875149 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e0014860 con 0x7f40f40f7d90 2021-11-20T08:59:40.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.181+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f410cc00 con 0x7f40f40f7d90 2021-11-20T08:59:40.187 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.182+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f4027a00 con 0x7f40f40f7d90 2021-11-20T08:59:40.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.182+0000 7f40f17fa700 1 -- 172.21.15.93:0/517875149 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f40e001b060 con 0x7f40f40f7d90 2021-11-20T08:59:40.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.183+0000 7f40f17fa700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f40e405c2a0 0x7f40e405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:40.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.183+0000 7f40f17fa700 1 -- 172.21.15.93:0/517875149 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f40e00760e0 con 0x7f40f40f7d90 2021-11-20T08:59:40.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.183+0000 7f40f8c60700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f40e405c2a0 0x7f40e405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:40.189 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.183+0000 7f40f8c60700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f40e405c2a0 0x7f40e405e760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f40e8005010 tx=0x7f40e8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:40.189 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.186+0000 7f40f17fa700 1 -- 172.21.15.93:0/517875149 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40e0048a40 con 0x7f40f40f7d90 2021-11-20T08:59:40.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.377+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40f40fef40 con 0x7f40f40f7d90 2021-11-20T08:59:40.377 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.378+0000 7f40f17fa700 1 -- 172.21.15.93:0/517875149 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f40e004c060 con 0x7f40f40f7d90 2021-11-20T08:59:40.378 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:40.378 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:40.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.379+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f40e405c2a0 msgr2=0x7f40e405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:40.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.379+0000 7f40faec4700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f40e405c2a0 0x7f40e405e760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f40e8005010 tx=0x7f40e8009b30).stop 2021-11-20T08:59:40.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.379+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 msgr2=0x7f40f41074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:40.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.379+0000 7f40faec4700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f41074b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f40e0002700 tx=0x7f40e0009d80).stop 2021-11-20T08:59:40.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.380+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 shutdown_connections 2021-11-20T08:59:40.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.380+0000 7f40faec4700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f40e405c2a0 0x7f40e405e760 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:40.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.380+0000 7f40faec4700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40f40f4780 0x7f40f4106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:40.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.380+0000 7f40faec4700 1 --2- 172.21.15.93:0/517875149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40f40f7d90 0x7f40f41074b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:40.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.380+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 >> 172.21.15.93:0/517875149 conn(0x7f40f40f04a0 msgr2=0x7f40f40fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:40.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.380+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 shutdown_connections 2021-11-20T08:59:40.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:40.380+0000 7f40faec4700 1 -- 172.21.15.93:0/517875149 wait complete. 2021-11-20T08:59:40.384 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:40.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:40.391 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:40.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:40 smithi093 conmon[33522]: cluster 2021-11-20T08:59:39.564342+0000 mgr.smithi093.qdikkm (mgr.14162) 273 : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:40.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:40 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:59:40.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:40 smithi093 conmon[33522]: 08:59:40.378985+0000 mon.smithi093 (mon.0) 728 : audit [DBG] from='client.? 172.21.15.93:0/517875149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:41.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:40 smithi102 conmon[36472]: cluster 2021-11-20T08:59:39.564342+0000 mgr.smithi093.qdikkm (mgr.14162) 273 : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:41.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:40 smithi102 conmon[36472]: audit 2021-11-20T08:59:40.378985+0000 mon.smithi093 (mon.0) 728 : audit [DBG] from='client.? 172.21.15.93:0/517875149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:41.392 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:41.524 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 -- 172.21.15.93:0/184169327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 msgr2=0x7f4b500f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:41.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 --2- 172.21.15.93:0/184169327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b500f6f90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f4b40004660 tx=0x7f4b40009b30).stop 2021-11-20T08:59:41.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 -- 172.21.15.93:0/184169327 shutdown_connections 2021-11-20T08:59:41.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 --2- 172.21.15.93:0/184169327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b500f5870 0x7f4b500f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:41.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 --2- 172.21.15.93:0/184169327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b500f6f90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:41.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 -- 172.21.15.93:0/184169327 >> 172.21.15.93:0/184169327 conn(0x7f4b500f0440 msgr2=0x7f4b500f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:41.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 -- 172.21.15.93:0/184169327 shutdown_connections 2021-11-20T08:59:41.527 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 -- 172.21.15.93:0/184169327 wait complete. 2021-11-20T08:59:41.527 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.522+0000 7f4b56348700 1 Processor -- start 2021-11-20T08:59:41.528 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b56348700 1 -- start start 2021-11-20T08:59:41.528 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b56348700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b500f5870 0x7f4b50105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:41.528 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b56348700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b501062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:41.528 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b56348700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50106860 con 0x7f4b500f6b70 2021-11-20T08:59:41.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b56348700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b501069a0 con 0x7f4b500f5870 2021-11-20T08:59:41.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b501062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:41.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b501062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43916/0 (socket says 172.21.15.93:43916) 2021-11-20T08:59:41.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4f7fe700 1 -- 172.21.15.93:0/1897932420 learned_addr learned my addr 172.21.15.93:0/1897932420 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:41.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4ffff700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b500f5870 0x7f4b50105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:41.530 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4f7fe700 1 -- 172.21.15.93:0/1897932420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b500f5870 msgr2=0x7f4b50105dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:41.530 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4f7fe700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b500f5870 0x7f4b50105dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:41.530 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4f7fe700 1 -- 172.21.15.93:0/1897932420 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40005040 con 0x7f4b500f6b70 2021-11-20T08:59:41.530 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4ffff700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b500f5870 0x7f4b50105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:59:41.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4f7fe700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b501062b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f4b44002700 tx=0x7f4b44009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:41.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4cff9700 1 -- 172.21.15.93:0/1897932420 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b44009320 con 0x7f4b500f6b70 2021-11-20T08:59:41.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5010b440 con 0x7f4b500f6b70 2021-11-20T08:59:41.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4cff9700 1 -- 172.21.15.93:0/1897932420 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b4400f040 con 0x7f4b500f6b70 2021-11-20T08:59:41.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.523+0000 7f4b4cff9700 1 -- 172.21.15.93:0/1897932420 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b44014900 con 0x7f4b500f6b70 2021-11-20T08:59:41.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.524+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5010ba00 con 0x7f4b500f6b70 2021-11-20T08:59:41.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.524+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b50027a00 con 0x7f4b500f6b70 2021-11-20T08:59:41.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.525+0000 7f4b4cff9700 1 -- 172.21.15.93:0/1897932420 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4b44009480 con 0x7f4b500f6b70 2021-11-20T08:59:41.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.525+0000 7f4b4cff9700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4b3805c2a0 0x7f4b3805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:41.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.525+0000 7f4b4cff9700 1 -- 172.21.15.93:0/1897932420 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f4b44076580 con 0x7f4b500f6b70 2021-11-20T08:59:41.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.525+0000 7f4b4ffff700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4b3805c2a0 0x7f4b3805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:41.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.525+0000 7f4b4ffff700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4b3805c2a0 0x7f4b3805e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4b40004e30 tx=0x7f4b40009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:41.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.529+0000 7f4b4cff9700 1 -- 172.21.15.93:0/1897932420 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b44048ee0 con 0x7f4b500f6b70 2021-11-20T08:59:41.714 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.715+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b500fdd60 con 0x7f4b500f6b70 2021-11-20T08:59:41.714 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.716+0000 7f4b4cff9700 1 -- 172.21.15.93:0/1897932420 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f4b4404c500 con 0x7f4b500f6b70 2021-11-20T08:59:41.715 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:41.715 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:41.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4b3805c2a0 msgr2=0x7f4b3805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:41.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4b3805c2a0 0x7f4b3805e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4b40004e30 tx=0x7f4b40009b30).stop 2021-11-20T08:59:41.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 msgr2=0x7f4b501062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:41.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b501062b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f4b44002700 tx=0x7f4b44009d80).stop 2021-11-20T08:59:41.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 shutdown_connections 2021-11-20T08:59:41.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4b3805c2a0 0x7f4b3805e760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:41.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b500f5870 0x7f4b50105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:41.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 --2- 172.21.15.93:0/1897932420 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b500f6b70 0x7f4b501062b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:41.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 >> 172.21.15.93:0/1897932420 conn(0x7f4b500f0440 msgr2=0x7f4b500fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:41.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 shutdown_connections 2021-11-20T08:59:41.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:41.717+0000 7f4b56348700 1 -- 172.21.15.93:0/1897932420 wait complete. 2021-11-20T08:59:41.720 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:41.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:41.730 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:42.731 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:42.859 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 -- 172.21.15.93:0/2471728294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 msgr2=0x7f2ba0069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:42.860 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/2471728294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0069ec0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f2b9c004660 tx=0x7f2b9c009b30).stop 2021-11-20T08:59:42.861 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 -- 172.21.15.93:0/2471728294 shutdown_connections 2021-11-20T08:59:42.861 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/2471728294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ba006a3b0 0x7f2ba006a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:42.861 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/2471728294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0069ec0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:42.861 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 -- 172.21.15.93:0/2471728294 >> 172.21.15.93:0/2471728294 conn(0x7f2ba00f1d10 msgr2=0x7f2ba00f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:42.862 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 -- 172.21.15.93:0/2471728294 shutdown_connections 2021-11-20T08:59:42.863 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.856+0000 7f2ba68b5700 1 -- 172.21.15.93:0/2471728294 wait complete. 2021-11-20T08:59:42.863 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba68b5700 1 Processor -- start 2021-11-20T08:59:42.863 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba68b5700 1 -- start start 2021-11-20T08:59:42.863 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba68b5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ba006a3b0 0x7f2ba0108f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:42.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba68b5700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0109470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:42.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba68b5700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba0109a20 con 0x7f2ba006b9c0 2021-11-20T08:59:42.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba68b5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba0109b60 con 0x7f2ba006a3b0 2021-11-20T08:59:42.864 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba50b2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0109470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:42.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba50b2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0109470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43926/0 (socket says 172.21.15.93:43926) 2021-11-20T08:59:42.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba50b2700 1 -- 172.21.15.93:0/3553637199 learned_addr learned my addr 172.21.15.93:0/3553637199 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:42.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.857+0000 7f2ba58b3700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ba006a3b0 0x7f2ba0108f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:42.865 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2ba50b2700 1 -- 172.21.15.93:0/3553637199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ba006a3b0 msgr2=0x7f2ba0108f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:42.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2ba50b2700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ba006a3b0 0x7f2ba0108f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:42.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2ba50b2700 1 -- 172.21.15.93:0/3553637199 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c005040 con 0x7f2ba006b9c0 2021-11-20T08:59:42.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2ba50b2700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0109470 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f2b900026d0 tx=0x7f2b90009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:42.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2b967fc700 1 -- 172.21.15.93:0/3553637199 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b900079c0 con 0x7f2ba006b9c0 2021-11-20T08:59:42.867 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2b967fc700 1 -- 172.21.15.93:0/3553637199 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b9000f040 con 0x7f2ba006b9c0 2021-11-20T08:59:42.868 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba010a500 con 0x7f2ba006b9c0 2021-11-20T08:59:42.868 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2b967fc700 1 -- 172.21.15.93:0/3553637199 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b900148b0 con 0x7f2ba006b9c0 2021-11-20T08:59:42.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.858+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba010aac0 con 0x7f2ba006b9c0 2021-11-20T08:59:42.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.859+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba0045ad0 con 0x7f2ba006b9c0 2021-11-20T08:59:42.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.859+0000 7f2b967fc700 1 -- 172.21.15.93:0/3553637199 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2b90007b20 con 0x7f2ba006b9c0 2021-11-20T08:59:42.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.859+0000 7f2b967fc700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2b8c05c300 0x7f2b8c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:42.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.860+0000 7f2b967fc700 1 -- 172.21.15.93:0/3553637199 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f2b900761a0 con 0x7f2ba006b9c0 2021-11-20T08:59:42.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.860+0000 7f2ba58b3700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2b8c05c300 0x7f2b8c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:42.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.860+0000 7f2ba58b3700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2b8c05c300 0x7f2b8c05e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2b9c009f50 tx=0x7f2b9c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:42.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:42.864+0000 7f2b967fc700 1 -- 172.21.15.93:0/3553637199 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b90048b00 con 0x7f2ba006b9c0 2021-11-20T08:59:42.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:42 smithi093 conmon[33522]: cluster 2021-11-20T08:59:41.564756+0000 mgr.smithi093.qdikkm (mgr.14162) 274 : cluster [DBG] 2021-11-20T08:59:42.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:42 smithi093 conmon[33522]: pgmap v247: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:42.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:42 smithi093 conmon[33522]: audit 2021-11-20T08:59:41.716654+0000 mon.smithi093 (mon.0) 729 : audit [DBG] from='client.? 172.21.15.93:0/1897932420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:43.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.048+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ba0027a00 con 0x7f2ba006b9c0 2021-11-20T08:59:43.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.049+0000 7f2b967fc700 1 -- 172.21.15.93:0/3553637199 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f2b9004c120 con 0x7f2ba006b9c0 2021-11-20T08:59:43.048 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:43.048 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:43.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2b8c05c300 msgr2=0x7f2b8c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:43.050 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2b8c05c300 0x7f2b8c05e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2b9c009f50 tx=0x7f2b9c009b30).stop 2021-11-20T08:59:43.051 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 msgr2=0x7f2ba0109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:43.051 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0109470 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f2b900026d0 tx=0x7f2b90009de0).stop 2021-11-20T08:59:43.051 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 shutdown_connections 2021-11-20T08:59:43.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2b8c05c300 0x7f2b8c05e7c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:43.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ba006a3b0 0x7f2ba0108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:43.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 --2- 172.21.15.93:0/3553637199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ba006b9c0 0x7f2ba0109470 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:43.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 >> 172.21.15.93:0/3553637199 conn(0x7f2ba00f1d10 msgr2=0x7f2ba00fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:43.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 shutdown_connections 2021-11-20T08:59:43.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:43.050+0000 7f2ba68b5700 1 -- 172.21.15.93:0/3553637199 wait complete. 2021-11-20T08:59:43.054 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:43.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:43.063 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:43.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:42 smithi102 conmon[36472]: cluster 2021-11-20T08:59:41.564756+0000 mgr.smithi093.qdikkm (mgr.14162) 274 : 2021-11-20T08:59:43.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:42 smithi102 conmon[36472]: cluster [DBG] pgmap v247: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:43.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:42 smithi102 conmon[36472]: audit 2021-11-20T08:59:41.716654+0000 mon.smithi093 (mon.0) 729 : audit [DBG] from='client.? 172.21.15.93:0/1897932420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:43.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:43 smithi093 conmon[33522]: audit 2021-11-20T08:59:43.049616+0000 mon.smithi093 (mon.0) 2021-11-20T08:59:43.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:43 smithi093 conmon[33522]: 730 : audit [DBG] from='client.? 172.21.15.93:0/3553637199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:44.064 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:44.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:43 smithi102 conmon[36472]: audit 2021-11-20T08:59:43.049616+0000 mon.smithi093 (mon.0 2021-11-20T08:59:44.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:43 smithi102 conmon[36472]: ) 730 : audit [DBG] from='client.? 172.21.15.93:0/3553637199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:44.193 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 -- 172.21.15.93:0/1192402181 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 msgr2=0x7f7e7c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:44.194 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/1192402181 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c0f6bd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f7e70004660 tx=0x7f7e70009b30).stop 2021-11-20T08:59:44.194 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 -- 172.21.15.93:0/1192402181 shutdown_connections 2021-11-20T08:59:44.194 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/1192402181 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c0f6bd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:44.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/1192402181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e7c0f54d0 0x7f7e7c0f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:44.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 -- 172.21.15.93:0/1192402181 >> 172.21.15.93:0/1192402181 conn(0x7f7e7c0f0c30 msgr2=0x7f7e7c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:44.195 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 -- 172.21.15.93:0/1192402181 shutdown_connections 2021-11-20T08:59:44.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.189+0000 7f7e7bfff700 1 -- 172.21.15.93:0/1192402181 wait complete. 2021-11-20T08:59:44.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7bfff700 1 Processor -- start 2021-11-20T08:59:44.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7bfff700 1 -- start start 2021-11-20T08:59:44.196 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e7c0f54d0 0x7f7e7c108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:44.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:44.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7bfff700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c1099e0 con 0x7f7e7c0f6770 2021-11-20T08:59:44.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7bfff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c109b20 con 0x7f7e7c0f54d0 2021-11-20T08:59:44.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7a7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:44.197 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7a7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43936/0 (socket says 172.21.15.93:43936) 2021-11-20T08:59:44.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7a7fc700 1 -- 172.21.15.93:0/869645003 learned_addr learned my addr 172.21.15.93:0/869645003 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:44.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.190+0000 7f7e7affd700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e7c0f54d0 0x7f7e7c108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:44.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e7a7fc700 1 -- 172.21.15.93:0/869645003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e7c0f54d0 msgr2=0x7f7e7c108f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:44.198 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e7a7fc700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e7c0f54d0 0x7f7e7c108f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:44.199 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e7a7fc700 1 -- 172.21.15.93:0/869645003 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70005040 con 0x7f7e7c0f6770 2021-11-20T08:59:44.202 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e7a7fc700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c109430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f7e70009830 tx=0x7f7e70009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:44.202 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e637fe700 1 -- 172.21.15.93:0/869645003 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e70018070 con 0x7f7e7c0f6770 2021-11-20T08:59:44.202 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c10a460 con 0x7f7e7c0f6770 2021-11-20T08:59:44.203 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e637fe700 1 -- 172.21.15.93:0/869645003 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e70013410 con 0x7f7e7c0f6770 2021-11-20T08:59:44.203 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e637fe700 1 -- 172.21.15.93:0/869645003 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e7001c400 con 0x7f7e7c0f6770 2021-11-20T08:59:44.203 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.191+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c10a9c0 con 0x7f7e7c0f6770 2021-11-20T08:59:44.203 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.192+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c045ad0 con 0x7f7e7c0f6770 2021-11-20T08:59:44.204 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.192+0000 7f7e637fe700 1 -- 172.21.15.93:0/869645003 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7e7001d420 con 0x7f7e7c0f6770 2021-11-20T08:59:44.204 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.192+0000 7f7e637fe700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e6405c300 0x7f7e6405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:44.204 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.192+0000 7f7e637fe700 1 -- 172.21.15.93:0/869645003 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f7e700780a0 con 0x7f7e7c0f6770 2021-11-20T08:59:44.204 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.193+0000 7f7e7affd700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e6405c300 0x7f7e6405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:44.205 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.193+0000 7f7e7affd700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e6405c300 0x7f7e6405e7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7e7c0f65d0 tx=0x7f7e6c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:44.205 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.196+0000 7f7e637fe700 1 -- 172.21.15.93:0/869645003 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e7004aa00 con 0x7f7e7c0f6770 2021-11-20T08:59:44.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.386+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e7c0fdbf0 con 0x7f7e7c0f6770 2021-11-20T08:59:44.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.386+0000 7f7e637fe700 1 -- 172.21.15.93:0/869645003 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f7e7004e020 con 0x7f7e7c0f6770 2021-11-20T08:59:44.386 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:44.386 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:44.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e6405c300 msgr2=0x7f7e6405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:44.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e6405c300 0x7f7e6405e7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7e7c0f65d0 tx=0x7f7e6c009040).stop 2021-11-20T08:59:44.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 msgr2=0x7f7e7c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:44.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c109430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f7e70009830 tx=0x7f7e70009b30).stop 2021-11-20T08:59:44.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 shutdown_connections 2021-11-20T08:59:44.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e6405c300 0x7f7e6405e7c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:44.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e7c0f54d0 0x7f7e7c108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:44.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 --2- 172.21.15.93:0/869645003 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e7c0f6770 0x7f7e7c109430 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:44.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 >> 172.21.15.93:0/869645003 conn(0x7f7e7c0f0c30 msgr2=0x7f7e7c05f780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:44.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 shutdown_connections 2021-11-20T08:59:44.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:44.388+0000 7f7e7bfff700 1 -- 172.21.15.93:0/869645003 wait complete. 2021-11-20T08:59:44.391 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:44.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:44.400 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:44.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:44 smithi093 conmon[33522]: cluster 2021-11-20T08:59:43.565238+0000 mgr.smithi093.qdikkm (mgr.14162) 275 : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 181 B/s wr, 1 op/s 2021-11-20T08:59:44.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:44 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T08:59:44.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:44 smithi093 conmon[33522]: 08:59:44.387105+0000 mon.smithi093 (mon.0) 731 : audit [DBG] from='client.? 172.21.15.93:0/869645003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:45.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:44 smithi102 conmon[36472]: cluster 2021-11-20T08:59:43.565238+0000 mgr.smithi093.qdikkm (mgr.14162) 275 : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 181 B/s wr, 1 op/s 2021-11-20T08:59:45.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:44 smithi102 conmon[36472]: audit 2021-11-20T08:59:44.387105+0000 mon.smithi093 (mon.0) 731 : audit [DBG] from='client.? 172.21.15.93:0/869645003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:45.401 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:45.530 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 -- 172.21.15.93:0/4275207347 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f4860 msgr2=0x7f81c40f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:45.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 --2- 172.21.15.93:0/4275207347 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f4860 0x7f81c40f4c80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f81b4004660 tx=0x7f81b4009b30).stop 2021-11-20T08:59:45.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 -- 172.21.15.93:0/4275207347 shutdown_connections 2021-11-20T08:59:45.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 --2- 172.21.15.93:0/4275207347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81c40f5200 0x7f81c40f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:45.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 --2- 172.21.15.93:0/4275207347 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f4860 0x7f81c40f4c80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:45.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 -- 172.21.15.93:0/4275207347 >> 172.21.15.93:0/4275207347 conn(0x7f81c40f0360 msgr2=0x7f81c40f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:45.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 -- 172.21.15.93:0/4275207347 shutdown_connections 2021-11-20T08:59:45.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.528+0000 7f81c3fff700 1 -- 172.21.15.93:0/4275207347 wait complete. 2021-11-20T08:59:45.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c3fff700 1 Processor -- start 2021-11-20T08:59:45.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c3fff700 1 -- start start 2021-11-20T08:59:45.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81c40f4860 0x7f81c4106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:45.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c3fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f5200 0x7f81c4107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:45.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c3fff700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c41079d0 con 0x7f81c40f5200 2021-11-20T08:59:45.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c3fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c4108140 con 0x7f81c40f4860 2021-11-20T08:59:45.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c27fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f5200 0x7f81c4107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:45.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c27fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f5200 0x7f81c4107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43946/0 (socket says 172.21.15.93:43946) 2021-11-20T08:59:45.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c27fc700 1 -- 172.21.15.93:0/3465377727 learned_addr learned my addr 172.21.15.93:0/3465377727 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:45.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c2ffd700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81c40f4860 0x7f81c4106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:45.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c27fc700 1 -- 172.21.15.93:0/3465377727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81c40f4860 msgr2=0x7f81c4106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:45.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c27fc700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81c40f4860 0x7f81c4106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:45.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c27fc700 1 -- 172.21.15.93:0/3465377727 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b4005040 con 0x7f81c40f5200 2021-11-20T08:59:45.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.529+0000 7f81c27fc700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f5200 0x7f81c4107390 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f81b80093f0 tx=0x7f81b8006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:45.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.530+0000 7f81ab7fe700 1 -- 172.21.15.93:0/3465377727 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81b8009ea0 con 0x7f81c40f5200 2021-11-20T08:59:45.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.530+0000 7f81ab7fe700 1 -- 172.21.15.93:0/3465377727 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81b8013af0 con 0x7f81c40f5200 2021-11-20T08:59:45.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.530+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c4108420 con 0x7f81c40f5200 2021-11-20T08:59:45.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.530+0000 7f81ab7fe700 1 -- 172.21.15.93:0/3465377727 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81b800dad0 con 0x7f81c40f5200 2021-11-20T08:59:45.537 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.530+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c41089e0 con 0x7f81c40f5200 2021-11-20T08:59:45.537 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.531+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c40f4010 con 0x7f81c40f5200 2021-11-20T08:59:45.537 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.531+0000 7f81ab7fe700 1 -- 172.21.15.93:0/3465377727 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f81b801a060 con 0x7f81c40f5200 2021-11-20T08:59:45.537 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.531+0000 7f81ab7fe700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f81ac05c300 0x7f81ac05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:45.537 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.531+0000 7f81ab7fe700 1 -- 172.21.15.93:0/3465377727 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f81b8076290 con 0x7f81c40f5200 2021-11-20T08:59:45.538 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.531+0000 7f81c2ffd700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f81ac05c300 0x7f81ac05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:45.538 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.532+0000 7f81c2ffd700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f81ac05c300 0x7f81ac05e7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f81b4009d60 tx=0x7f81b4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:45.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.535+0000 7f81ab7fe700 1 -- 172.21.15.93:0/3465377727 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81b8048b90 con 0x7f81c40f5200 2021-11-20T08:59:45.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.720+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81c40fee90 con 0x7f81c40f5200 2021-11-20T08:59:45.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.720+0000 7f81ab7fe700 1 -- 172.21.15.93:0/3465377727 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f81b804c1b0 con 0x7f81c40f5200 2021-11-20T08:59:45.720 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:45.720 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:45.721 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f81ac05c300 msgr2=0x7f81ac05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:45.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f81ac05c300 0x7f81ac05e7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f81b4009d60 tx=0x7f81b4009b30).stop 2021-11-20T08:59:45.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f5200 msgr2=0x7f81c4107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:45.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f5200 0x7f81c4107390 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f81b80093f0 tx=0x7f81b8006ec0).stop 2021-11-20T08:59:45.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 shutdown_connections 2021-11-20T08:59:45.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f81ac05c300 0x7f81ac05e7c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:45.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81c40f4860 0x7f81c4106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:45.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 --2- 172.21.15.93:0/3465377727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81c40f5200 0x7f81c4107390 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:45.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 >> 172.21.15.93:0/3465377727 conn(0x7f81c40f0360 msgr2=0x7f81c40fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:45.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 shutdown_connections 2021-11-20T08:59:45.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:45.722+0000 7f81c3fff700 1 -- 172.21.15.93:0/3465377727 wait complete. 2021-11-20T08:59:45.724 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:45.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:45.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:46.736 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:46.882 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 -- 172.21.15.93:0/2865856559 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f53a0 msgr2=0x7fd2940f7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:46.883 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/2865856559 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f53a0 0x7fd2940f7ce0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fd284004660 tx=0x7fd284009b30).stop 2021-11-20T08:59:46.883 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 -- 172.21.15.93:0/2865856559 shutdown_connections 2021-11-20T08:59:46.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/2865856559 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f53a0 0x7fd2940f7ce0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:46.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/2865856559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2940f4a00 0x7fd2940f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:46.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 -- 172.21.15.93:0/2865856559 >> 172.21.15.93:0/2865856559 conn(0x7fd2940f0440 msgr2=0x7fd2940f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:46.889 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 -- 172.21.15.93:0/2865856559 shutdown_connections 2021-11-20T08:59:46.889 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.871+0000 7fd29c5c9700 1 -- 172.21.15.93:0/2865856559 wait complete. 2021-11-20T08:59:46.890 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29c5c9700 1 Processor -- start 2021-11-20T08:59:46.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29c5c9700 1 -- start start 2021-11-20T08:59:46.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29c5c9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f4a00 0x7fd294107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:46.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29c5c9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2940f53a0 0x7fd294107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:46.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29c5c9700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd294107ad0 con 0x7fd2940f4a00 2021-11-20T08:59:46.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29c5c9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd294107c10 con 0x7fd2940f53a0 2021-11-20T08:59:46.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29a365700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f4a00 0x7fd294107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:46.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29a365700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f4a00 0x7fd294107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43954/0 (socket says 172.21.15.93:43954) 2021-11-20T08:59:46.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29a365700 1 -- 172.21.15.93:0/3399684963 learned_addr learned my addr 172.21.15.93:0/3399684963 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:46.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29a365700 1 -- 172.21.15.93:0/3399684963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2940f53a0 msgr2=0x7fd294107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:46.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29a365700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2940f53a0 0x7fd294107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:46.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29a365700 1 -- 172.21.15.93:0/3399684963 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284005040 con 0x7fd2940f4a00 2021-11-20T08:59:46.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29a365700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f4a00 0x7fd294107030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fd29000d8a0 tx=0x7fd290004f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:46.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd28affd700 1 -- 172.21.15.93:0/3399684963 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd29000b5a0 con 0x7fd2940f4a00 2021-11-20T08:59:46.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd28affd700 1 -- 172.21.15.93:0/3399684963 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd29000f040 con 0x7fd2940f4a00 2021-11-20T08:59:46.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.872+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29410c6b0 con 0x7fd2940f4a00 2021-11-20T08:59:46.898 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.873+0000 7fd28affd700 1 -- 172.21.15.93:0/3399684963 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd290013b10 con 0x7fd2940f4a00 2021-11-20T08:59:46.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.873+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29410cc70 con 0x7fd2940f4a00 2021-11-20T08:59:46.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.873+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29405f860 con 0x7fd2940f4a00 2021-11-20T08:59:46.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.874+0000 7fd28affd700 1 -- 172.21.15.93:0/3399684963 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd29001a030 con 0x7fd2940f4a00 2021-11-20T08:59:46.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.874+0000 7fd28affd700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd28005c2a0 0x7fd28005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:46.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.874+0000 7fd28affd700 1 -- 172.21.15.93:0/3399684963 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fd290076000 con 0x7fd2940f4a00 2021-11-20T08:59:46.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.874+0000 7fd299b64700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd28005c2a0 0x7fd28005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:46.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.875+0000 7fd299b64700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd28005c2a0 0x7fd28005e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd284009f50 tx=0x7fd284009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:46.901 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:46.878+0000 7fd28affd700 1 -- 172.21.15.93:0/3399684963 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd290048a90 con 0x7fd2940f4a00 2021-11-20T08:59:46.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:46 smithi093 conmon[33522]: cluster 2021-11-20T08:59:45.565647+0000 mgr.smithi093.qdikkm (mgr.14162) 276 : cluster 2021-11-20T08:59:46.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:46 smithi093 conmon[33522]: [DBG] pgmap v249: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:59:46.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:46 smithi093 conmon[33522]: audit 2021-11-20T08:59:45.721331+0000 mon.smithi093 (mon.0) 732 : audit [DBG] from='client.? 172.21.15.93:0/3465377727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:47.064 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.063+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd294027a00 con 0x7fd2940f4a00 2021-11-20T08:59:47.065 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.064+0000 7fd28affd700 1 -- 172.21.15.93:0/3399684963 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7fd29004c0b0 con 0x7fd2940f4a00 2021-11-20T08:59:47.066 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:47.075 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:47.079 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.065+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd28005c2a0 msgr2=0x7fd28005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:47.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.065+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd28005c2a0 0x7fd28005e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd284009f50 tx=0x7fd284009b30).stop 2021-11-20T08:59:47.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.065+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f4a00 msgr2=0x7fd294107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:47.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.065+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f4a00 0x7fd294107030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fd29000d8a0 tx=0x7fd290004f00).stop 2021-11-20T08:59:47.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.066+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 shutdown_connections 2021-11-20T08:59:47.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.066+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd28005c2a0 0x7fd28005e760 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:47.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.066+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2940f4a00 0x7fd294107030 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:47.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.066+0000 7fd29c5c9700 1 --2- 172.21.15.93:0/3399684963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2940f53a0 0x7fd294107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:47.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.066+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 >> 172.21.15.93:0/3399684963 conn(0x7fd2940f0440 msgr2=0x7fd2940fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:47.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.066+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 shutdown_connections 2021-11-20T08:59:47.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:47.066+0000 7fd29c5c9700 1 -- 172.21.15.93:0/3399684963 wait complete. 2021-11-20T08:59:47.085 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:47.086 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:47.087 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:47.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:46 smithi102 conmon[36472]: cluster 2021-11-20T08:59:45.565647+0000 mgr.smithi093.qdikkm (mgr.14162) 276 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:59:47.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:46 smithi102 conmon[36472]: 2021-11-20T08:59:47.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:46 smithi102 conmon[36472]: audit 2021-11-20T08:59:45.721331+0000 mon.smithi093 (mon.0) 732 : audit [DBG] from='client.? 172.21.15.93:0/3465377727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:47.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:47 smithi093 conmon[33522]: audit 2021-11-20T08:59:47 2021-11-20T08:59:47.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:47 smithi093 conmon[33522]: .064988+0000 mon.smithi093 (mon.0) 733 : audit [DBG] from='client.? 172.21.15.93:0/3399684963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:48.089 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:48.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:47 smithi102 conmon[36472]: audit 2021-11-20T08:59:47.064988+0000 mon.smithi093 (mon.0) 733 : audit [DBG] from='client.? 172.21.15.93:0/3399684963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:48.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 -- 172.21.15.93:0/2096616918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 msgr2=0x7f27600f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:48.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 --2- 172.21.15.93:0/2096616918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27600f81b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f27500099e0 tx=0x7f2750004d10).stop 2021-11-20T08:59:48.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 -- 172.21.15.93:0/2096616918 shutdown_connections 2021-11-20T08:59:48.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 --2- 172.21.15.93:0/2096616918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27600f4780 0x7f27600f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:48.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 --2- 172.21.15.93:0/2096616918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27600f81b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:48.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 -- 172.21.15.93:0/2096616918 >> 172.21.15.93:0/2096616918 conn(0x7f27600f04a0 msgr2=0x7f27600f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:48.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 -- 172.21.15.93:0/2096616918 shutdown_connections 2021-11-20T08:59:48.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.216+0000 7f2767318700 1 -- 172.21.15.93:0/2096616918 wait complete. 2021-11-20T08:59:48.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f2767318700 1 Processor -- start 2021-11-20T08:59:48.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f2767318700 1 -- start start 2021-11-20T08:59:48.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f2767318700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27600f4780 0x7f27601070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:48.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f2767318700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27601075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:48.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f2767318700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2760107b50 con 0x7f27600f7d90 2021-11-20T08:59:48.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f2767318700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2760107c90 con 0x7f27600f4780 2021-11-20T08:59:48.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27648b3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27601075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:48.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27648b3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27601075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43966/0 (socket says 172.21.15.93:43966) 2021-11-20T08:59:48.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27648b3700 1 -- 172.21.15.93:0/572246528 learned_addr learned my addr 172.21.15.93:0/572246528 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:48.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27650b4700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27600f4780 0x7f27601070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:48.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27648b3700 1 -- 172.21.15.93:0/572246528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27600f4780 msgr2=0x7f27601070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:48.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27648b3700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27600f4780 0x7f27601070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:48.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27648b3700 1 -- 172.21.15.93:0/572246528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750009640 con 0x7f27600f7d90 2021-11-20T08:59:48.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.217+0000 7f27648b3700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27601075a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f2754002700 tx=0x7f2754009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:48.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.218+0000 7f275dffb700 1 -- 172.21.15.93:0/572246528 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27540079c0 con 0x7f27600f7d90 2021-11-20T08:59:48.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.218+0000 7f275dffb700 1 -- 172.21.15.93:0/572246528 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f275400f040 con 0x7f27600f7d90 2021-11-20T08:59:48.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.218+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276010c730 con 0x7f27600f7d90 2021-11-20T08:59:48.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.218+0000 7f275dffb700 1 -- 172.21.15.93:0/572246528 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2754014860 con 0x7f27600f7d90 2021-11-20T08:59:48.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.218+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276010cc70 con 0x7f27600f7d90 2021-11-20T08:59:48.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.219+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2760027a00 con 0x7f27600f7d90 2021-11-20T08:59:48.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.219+0000 7f275dffb700 1 -- 172.21.15.93:0/572246528 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f275401b060 con 0x7f27600f7d90 2021-11-20T08:59:48.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.219+0000 7f275dffb700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f274805c2a0 0x7f274805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:48.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.220+0000 7f275dffb700 1 -- 172.21.15.93:0/572246528 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f2754076090 con 0x7f27600f7d90 2021-11-20T08:59:48.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.220+0000 7f27650b4700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f274805c2a0 0x7f274805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:48.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.220+0000 7f27650b4700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f274805c2a0 0x7f274805e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f275000b620 tx=0x7f2750004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:48.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.223+0000 7f275dffb700 1 -- 172.21.15.93:0/572246528 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2754048a70 con 0x7f27600f7d90 2021-11-20T08:59:48.408 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.410+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27600fef40 con 0x7f27600f7d90 2021-11-20T08:59:48.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.410+0000 7f275dffb700 1 -- 172.21.15.93:0/572246528 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f275404c090 con 0x7f27600f7d90 2021-11-20T08:59:48.409 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:48.410 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:48.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f274805c2a0 msgr2=0x7f274805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:48.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f274805c2a0 0x7f274805e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f275000b620 tx=0x7f2750004d10).stop 2021-11-20T08:59:48.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 msgr2=0x7f27601075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:48.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27601075a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f2754002700 tx=0x7f2754009d80).stop 2021-11-20T08:59:48.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 shutdown_connections 2021-11-20T08:59:48.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f274805c2a0 0x7f274805e760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:48.413 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27600f4780 0x7f27601070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:48.413 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 --2- 172.21.15.93:0/572246528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27600f7d90 0x7f27601075a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:48.413 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 >> 172.21.15.93:0/572246528 conn(0x7f27600f04a0 msgr2=0x7f27600fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:48.413 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 shutdown_connections 2021-11-20T08:59:48.413 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:48.412+0000 7f2767318700 1 -- 172.21.15.93:0/572246528 wait complete. 2021-11-20T08:59:48.414 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:48.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:48.423 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:48 smithi093 conmon[33522]: cluster 2021-11-20T08:59:47.566096+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:59:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:48 smithi093 conmon[33522]: ) 277 : cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:59:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:48 smithi093 conmon[33522]: audit 2021-11-20T08:59:48.411199+0000 mon.smithi093 (mon.0) 734 : audit [DBG] from='client.? 172.21.15.93:0/572246528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:49.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:48 smithi102 conmon[36472]: cluster 2021-11-20T08:59:47.566096+0000 mgr.smithi093.qdikkm (mgr.14162) 277 : cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T08:59:49.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:48 smithi102 conmon[36472]: audit 2021-11-20T08:59:48.411199+0000 mon.smithi093 (mon.0) 734 : audit [DBG] from='client.? 172.21.15.93:0/572246528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:49.425 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:49.553 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 -- 172.21.15.93:0/105438727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 msgr2=0x7fc2f00f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:49.553 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/105438727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f00f6ca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc2e4004660 tx=0x7fc2e4009b30).stop 2021-11-20T08:59:49.553 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 -- 172.21.15.93:0/105438727 shutdown_connections 2021-11-20T08:59:49.554 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/105438727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f00f6ca0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:49.554 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/105438727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f00f55a0 0x7fc2f00f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:49.554 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 -- 172.21.15.93:0/105438727 >> 172.21.15.93:0/105438727 conn(0x7fc2f00f0ce0 msgr2=0x7fc2f00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:49.554 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 -- 172.21.15.93:0/105438727 shutdown_connections 2021-11-20T08:59:49.555 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 -- 172.21.15.93:0/105438727 wait complete. 2021-11-20T08:59:49.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 Processor -- start 2021-11-20T08:59:49.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.551+0000 7fc2f6d84700 1 -- start start 2021-11-20T08:59:49.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f6d84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f00f55a0 0x7fc2f005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:49.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f6d84700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:49.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f6d84700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0107a00 con 0x7fc2f00f6840 2021-11-20T08:59:49.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f6d84700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f010e510 con 0x7fc2f00f55a0 2021-11-20T08:59:49.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2effff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:49.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2effff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f010e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43976/0 (socket says 172.21.15.93:43976) 2021-11-20T08:59:49.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2effff700 1 -- 172.21.15.93:0/1140450898 learned_addr learned my addr 172.21.15.93:0/1140450898 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:49.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f4b20700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f00f55a0 0x7fc2f005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:49.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2effff700 1 -- 172.21.15.93:0/1140450898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f00f55a0 msgr2=0x7fc2f005ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:49.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2effff700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f00f55a0 0x7fc2f005ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:49.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2effff700 1 -- 172.21.15.93:0/1140450898 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e4005040 con 0x7fc2f00f6840 2021-11-20T08:59:49.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f4b20700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f00f55a0 0x7fc2f005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:59:49.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2effff700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f010e020 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fc2e4009870 tx=0x7fc2e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:49.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2ed7fa700 1 -- 172.21.15.93:0/1140450898 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e4019070 con 0x7fc2f00f6840 2021-11-20T08:59:49.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2ed7fa700 1 -- 172.21.15.93:0/1140450898 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2e4007e70 con 0x7fc2f00f6840 2021-11-20T08:59:49.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f010e650 con 0x7fc2f00f6840 2021-11-20T08:59:49.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2ed7fa700 1 -- 172.21.15.93:0/1140450898 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e4014d00 con 0x7fc2f00f6840 2021-11-20T08:59:49.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.552+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f010ebb0 con 0x7fc2f00f6840 2021-11-20T08:59:49.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.553+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0027a00 con 0x7fc2f00f6840 2021-11-20T08:59:49.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.554+0000 7fc2ed7fa700 1 -- 172.21.15.93:0/1140450898 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc2e4025070 con 0x7fc2f00f6840 2021-11-20T08:59:49.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.554+0000 7fc2ed7fa700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc2d805c2a0 0x7fc2d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:49.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.554+0000 7fc2ed7fa700 1 -- 172.21.15.93:0/1140450898 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fc2e4077260 con 0x7fc2f00f6840 2021-11-20T08:59:49.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.554+0000 7fc2f4b20700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc2d805c2a0 0x7fc2d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:49.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.555+0000 7fc2f4b20700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc2d805c2a0 0x7fc2d805e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc2f00f66a0 tx=0x7fc2e000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:49.567 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.558+0000 7fc2ed7fa700 1 -- 172.21.15.93:0/1140450898 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2e4049b60 con 0x7fc2f00f6840 2021-11-20T08:59:49.744 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.745+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2f00fdd30 con 0x7fc2f00f6840 2021-11-20T08:59:49.744 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.746+0000 7fc2ed7fa700 1 -- 172.21.15.93:0/1140450898 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7fc2e4023070 con 0x7fc2f00f6840 2021-11-20T08:59:49.745 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:49.745 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:49.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc2d805c2a0 msgr2=0x7fc2d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:49.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc2d805c2a0 0x7fc2d805e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc2f00f66a0 tx=0x7fc2e000b040).stop 2021-11-20T08:59:49.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 msgr2=0x7fc2f010e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:49.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f010e020 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fc2e4009870 tx=0x7fc2e400b040).stop 2021-11-20T08:59:49.751 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 shutdown_connections 2021-11-20T08:59:49.751 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc2d805c2a0 0x7fc2d805e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:49.752 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f00f55a0 0x7fc2f005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:49.752 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 --2- 172.21.15.93:0/1140450898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2f00f6840 0x7fc2f010e020 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:49.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 >> 172.21.15.93:0/1140450898 conn(0x7fc2f00f0ce0 msgr2=0x7fc2f00fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:49.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.747+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 shutdown_connections 2021-11-20T08:59:49.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:49.748+0000 7fc2f6d84700 1 -- 172.21.15.93:0/1140450898 wait complete. 2021-11-20T08:59:49.754 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:49.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:49.760 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:50.761 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:50.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 -- 172.21.15.93:0/3334795427 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f6840 msgr2=0x7f82300f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:50.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 --2- 172.21.15.93:0/3334795427 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f6840 0x7f82300f6ca0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f8218004660 tx=0x7f8218009b30).stop 2021-11-20T08:59:50.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 -- 172.21.15.93:0/3334795427 shutdown_connections 2021-11-20T08:59:50.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 --2- 172.21.15.93:0/3334795427 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f6840 0x7f82300f6ca0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:50.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 --2- 172.21.15.93:0/3334795427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82300f55a0 0x7f82300f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:50.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 -- 172.21.15.93:0/3334795427 >> 172.21.15.93:0/3334795427 conn(0x7f82300f0ce0 msgr2=0x7f82300f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:50.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 -- 172.21.15.93:0/3334795427 shutdown_connections 2021-11-20T08:59:50.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 -- 172.21.15.93:0/3334795427 wait complete. 2021-11-20T08:59:50.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.890+0000 7f8235a42700 1 Processor -- start 2021-11-20T08:59:50.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f8235a42700 1 -- start start 2021-11-20T08:59:50.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f8235a42700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f55a0 0x7f82301070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:50.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f8235a42700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82300f6840 0x7f82301075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:50.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f8235a42700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230107b50 con 0x7f82300f55a0 2021-11-20T08:59:50.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f8235a42700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230107c90 con 0x7f82300f6840 2021-11-20T08:59:50.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f55a0 0x7f82301070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:50.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f55a0 0x7f82301070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:43982/0 (socket says 172.21.15.93:43982) 2021-11-20T08:59:50.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822effd700 1 -- 172.21.15.93:0/2111348490 learned_addr learned my addr 172.21.15.93:0/2111348490 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:50.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822e7fc700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82300f6840 0x7f82301075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:50.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822effd700 1 -- 172.21.15.93:0/2111348490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82300f6840 msgr2=0x7f82301075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:50.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822effd700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82300f6840 0x7f82301075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:50.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822effd700 1 -- 172.21.15.93:0/2111348490 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8218005040 con 0x7f82300f55a0 2021-11-20T08:59:50.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f822effd700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f55a0 0x7f82301070b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f822000c760 tx=0x7f8220004cf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:50.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f82277fe700 1 -- 172.21.15.93:0/2111348490 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82200106f0 con 0x7f82300f55a0 2021-11-20T08:59:50.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f82277fe700 1 -- 172.21.15.93:0/2111348490 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8220010d00 con 0x7f82300f55a0 2021-11-20T08:59:50.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823010c730 con 0x7f82300f55a0 2021-11-20T08:59:50.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.891+0000 7f82277fe700 1 -- 172.21.15.93:0/2111348490 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8220012d10 con 0x7f82300f55a0 2021-11-20T08:59:50.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.892+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823010cc70 con 0x7f82300f55a0 2021-11-20T08:59:50.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.893+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82300fdd50 con 0x7f82300f55a0 2021-11-20T08:59:50.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.893+0000 7f82277fe700 1 -- 172.21.15.93:0/2111348490 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8220010070 con 0x7f82300f55a0 2021-11-20T08:59:50.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.893+0000 7f82277fe700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f821c05c2a0 0x7f821c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:50.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.893+0000 7f82277fe700 1 -- 172.21.15.93:0/2111348490 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f82200761e0 con 0x7f82300f55a0 2021-11-20T08:59:50.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.893+0000 7f822e7fc700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f821c05c2a0 0x7f821c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:50.901 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.894+0000 7f822e7fc700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f821c05c2a0 0x7f821c05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8218009d60 tx=0x7f8218009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:50.901 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:50.897+0000 7f82277fe700 1 -- 172.21.15.93:0/2111348490 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8220048b40 con 0x7f82300f55a0 2021-11-20T08:59:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:50 smithi093 conmon[33522]: cluster 2021-11-20T08:59:49.566464+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T08:59:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:50 smithi093 conmon[33522]: ) 278 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:59:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:50 smithi093 conmon[33522]: audit 2021-11-20T08:59:49.746585+0000 mon.smithi093 (mon.0) 735 2021-11-20T08:59:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:50 smithi093 conmon[33522]: : audit [DBG] from='client.? 172.21.15.93:0/1140450898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:51.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.081+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f823010c8c0 con 0x7f82300f55a0 2021-11-20T08:59:51.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.081+0000 7f82277fe700 1 -- 172.21.15.93:0/2111348490 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f823010c8c0 con 0x7f82300f55a0 2021-11-20T08:59:51.081 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:51.081 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":65},{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:35.663974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14590":{"gid":14590,"name":"cephfs.smithi093.zrryyo","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.93:6829/2300988268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":2300988268},{"type":"v1","addr":"172.21.15.93:6829","nonce":2300988268}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:51.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f821c05c2a0 msgr2=0x7f821c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:51.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f821c05c2a0 0x7f821c05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8218009d60 tx=0x7f8218009b30).stop 2021-11-20T08:59:51.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f55a0 msgr2=0x7f82301070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:51.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f55a0 0x7f82301070b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f822000c760 tx=0x7f8220004cf0).stop 2021-11-20T08:59:51.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 shutdown_connections 2021-11-20T08:59:51.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f821c05c2a0 0x7f821c05e760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:51.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f82300f55a0 0x7f82301070b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:51.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 --2- 172.21.15.93:0/2111348490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82300f6840 0x7f82301075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:51.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 >> 172.21.15.93:0/2111348490 conn(0x7f82300f0ce0 msgr2=0x7f82300f2840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:51.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 shutdown_connections 2021-11-20T08:59:51.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:51.083+0000 7f8235a42700 1 -- 172.21.15.93:0/2111348490 wait complete. 2021-11-20T08:59:51.085 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 65 2021-11-20T08:59:51.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:35.663974+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14590': {'gid': 14590, 'name': 'cephfs.smithi093.zrryyo', 'rank': 0, 'incarnation': 61, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.93:6829/2300988268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6828', 'nonce': 2300988268}, {'type': 'v1', 'addr': '172.21.15.93:6829', 'nonce': 2300988268}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:51.094 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14590:up:clientreplay 2021-11-20T08:59:51.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:50 smithi102 conmon[36472]: cluster 2021-11-20T08:59:49.566464+0000 mgr.smithi093.qdikkm (mgr.14162) 278 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:59:51.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:50 smithi102 conmon[36472]: audit 2021-11-20T08:59:49.746585+0000 mon.smithi093 (mon.0) 735 : audit [DBG] from='client.? 172.21.15.93:0/1140450898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:52.096 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:52.123 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: audit 2021-11-20T08:59:51.082158+0000 mon.smithi093 (mon.0) 736 : 2021-11-20T08:59:52.123 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: audit [DBG] from='client.? 172.21.15.93:0/2111348490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:52.123 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: cluster 2021-11-20T08:59:51.590944+0000 mon.smithi093 (mon.0) 737 : cluster [WRN] Replacing daemon mds.cephfs.smithi093.zrryyo as rank 0 with standby daemon mds.cephfs.smithi093.ljzugc 2021-11-20T08:59:52.124 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: cluster 2021-11-20T08: 2021-11-20T08:59:52.124 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: 59:51.591037+0000 mon.smithi093 (mon.0) 738 : cluster [INF] MDS daemon mds.cephfs.smithi093.zrryyo is removed because it is dead or otherwise unavailable. 2021-11-20T08:59:52.124 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: cluster 2021-11-20T08:59:51.595649+0000 2021-11-20T08:59:52.124 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: mon.smithi093 (mon.0) 739 : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T08:59:52.125 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: 2021-11-20T08:59:52.125 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: cluster 2021-11-20T08:59:52.125 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: 2021-11-20T08:59:51.598453+0000 mon.smithi093 ( 2021-11-20T08:59:52.125 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:51 smithi093 conmon[33522]: mon.0) 740 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 2 up:standby 2021-11-20T08:59:52.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: audit 2021-11-20T08:59:51.082158+0000 mon.smithi093 (mon.0) 736 : audit [DBG] from='client.? 172.21.15.93:0/2111348490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:52.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: cluster 2021-11-20T08:59:51.590944+0000 mon.smithi093 (mon.0) 737 : cluster [WRN] Replacing daemon mds.cephfs.smithi093.zrryyo as rank 0 with standby daemon mds.cephfs.smithi093.ljzugc 2021-11-20T08:59:52.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: cluster 2021-11-20T08:59:51.591037+0000 mon.smithi093 (mon.0) 738 : cluster [INF] MDS daemon mds.cephfs.smithi093.zrryyo is removed because it is dead or otherwise unavailable. 2021-11-20T08:59:52.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: cluster 2021-11-20T08:59:52.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: 2021-11-20T08:59:51.595649+0000 mon.smithi093 (mon.0) 739 : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T08:59:52.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: cluster 2021-11-20T08:59:51.598453 2021-11-20T08:59:52.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 740 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:replay} 2 up:standby 2021-11-20T08:59:52.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:51 smithi102 conmon[36472]: 2021-11-20T08:59:52.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 -- 172.21.15.93:0/3710548114 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f4a00 msgr2=0x7f66600f4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:52.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 --2- 172.21.15.93:0/3710548114 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f4a00 0x7f66600f4e20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6650004660 tx=0x7f6650009b30).stop 2021-11-20T08:59:52.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 -- 172.21.15.93:0/3710548114 shutdown_connections 2021-11-20T08:59:52.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 --2- 172.21.15.93:0/3710548114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600f53a0 0x7f66600f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:52.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 --2- 172.21.15.93:0/3710548114 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f4a00 0x7f66600f4e20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:52.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 -- 172.21.15.93:0/3710548114 >> 172.21.15.93:0/3710548114 conn(0x7f66600f0440 msgr2=0x7f66600f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:52.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 -- 172.21.15.93:0/3710548114 shutdown_connections 2021-11-20T08:59:52.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 -- 172.21.15.93:0/3710548114 wait complete. 2021-11-20T08:59:52.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 Processor -- start 2021-11-20T08:59:52.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.221+0000 7f66649e8700 1 -- start start 2021-11-20T08:59:52.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f66649e8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600f4a00 0x7f6660107090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:52.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f66649e8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f53a0 0x7f6660107580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:52.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f66649e8700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660107b30 con 0x7f66600f53a0 2021-11-20T08:59:52.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f66649e8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660107c70 con 0x7f66600f4a00 2021-11-20T08:59:52.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665dd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f53a0 0x7f6660107580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:52.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665dd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f53a0 0x7f6660107580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44004/0 (socket says 172.21.15.93:44004) 2021-11-20T08:59:52.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665dd9b700 1 -- 172.21.15.93:0/1857708751 learned_addr learned my addr 172.21.15.93:0/1857708751 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:52.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665e59c700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600f4a00 0x7f6660107090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:52.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665dd9b700 1 -- 172.21.15.93:0/1857708751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600f4a00 msgr2=0x7f6660107090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:52.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665dd9b700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600f4a00 0x7f6660107090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:52.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665dd9b700 1 -- 172.21.15.93:0/1857708751 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6650005040 con 0x7f66600f53a0 2021-11-20T08:59:52.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665e59c700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600f4a00 0x7f6660107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T08:59:52.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f665dd9b700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f53a0 0x7f6660107580 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f6654002700 tx=0x7f6654009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:52.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f664effd700 1 -- 172.21.15.93:0/1857708751 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6654009320 con 0x7f66600f53a0 2021-11-20T08:59:52.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666010c710 con 0x7f66600f53a0 2021-11-20T08:59:52.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f664effd700 1 -- 172.21.15.93:0/1857708751 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f665400f040 con 0x7f66600f53a0 2021-11-20T08:59:52.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f664effd700 1 -- 172.21.15.93:0/1857708751 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6654014900 con 0x7f66600f53a0 2021-11-20T08:59:52.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.222+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666010ccd0 con 0x7f66600f53a0 2021-11-20T08:59:52.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.223+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666005f860 con 0x7f66600f53a0 2021-11-20T08:59:52.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.224+0000 7f664effd700 1 -- 172.21.15.93:0/1857708751 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6654009480 con 0x7f66600f53a0 2021-11-20T08:59:52.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.224+0000 7f664effd700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f664805c2a0 0x7f664805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:52.231 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.224+0000 7f664effd700 1 -- 172.21.15.93:0/1857708751 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f66540765d0 con 0x7f66600f53a0 2021-11-20T08:59:52.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.224+0000 7f665e59c700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f664805c2a0 0x7f664805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:52.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.224+0000 7f665e59c700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f664805c2a0 0x7f664805e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f6650005010 tx=0x7f6650009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:52.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.228+0000 7f664effd700 1 -- 172.21.15.93:0/1857708751 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6654048ee0 con 0x7f66600f53a0 2021-11-20T08:59:52.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.411+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6660027a00 con 0x7f66600f53a0 2021-11-20T08:59:52.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.412+0000 7f664effd700 1 -- 172.21.15.93:0/1857708751 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2560 (secure 0 0 0) 0x7f665404c500 con 0x7f66600f53a0 2021-11-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:51.591551+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:replay","state_seq":3,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f664805c2a0 msgr2=0x7f664805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f664805c2a0 0x7f664805e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f6650005010 tx=0x7f6650009b30).stop 2021-11-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f53a0 msgr2=0x7f6660107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f53a0 0x7f6660107580 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f6654002700 tx=0x7f6654009d80).stop 2021-11-20T08:59:52.415 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 shutdown_connections 2021-11-20T08:59:52.415 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f664805c2a0 0x7f664805e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:52.415 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600f4a00 0x7f6660107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:52.416 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 --2- 172.21.15.93:0/1857708751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66600f53a0 0x7f6660107580 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:52.416 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 >> 172.21.15.93:0/1857708751 conn(0x7f66600f0440 msgr2=0x7f66600fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:52.416 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 shutdown_connections 2021-11-20T08:59:52.416 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:52.414+0000 7f66649e8700 1 -- 172.21.15.93:0/1857708751 wait complete. 2021-11-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 66 2021-11-20T08:59:52.424 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:51.591551+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:52.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:replay 2021-11-20T08:59:53.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: cluster 2021-11-20T08:59:51.567147+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T08:59:53.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: 279 : cluster [DBG] pgmap v252: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:59:53.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: audit 2021-11-20T08:59:52.413004+0000 mon.smithi093 (mon.0) 741 : audit [DBG] from='client.? 172.21.15.93:0/1857708751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:53.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: cluster 2021-11-20T08:59:53.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: 2021-11-20T08:59:52.600148+0000 mon.smithi093 (mon.0) 742 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:reconnect 2021-11-20T08:59:53.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: cluster 2021-11-20T08:59:52 2021-11-20T08:59:53.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: .600200+0000 mon.smithi093 (mon.0) 743 : cluster 2021-11-20T08:59:53.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:52 smithi102 conmon[36472]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 2 up:standby 2021-11-20T08:59:53.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: cluster 2021-11-20T08:59:51.567147+0000 mgr.smithi093.qdikkm ( 2021-11-20T08:59:53.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: mgr.14162) 279 : cluster [DBG] pgmap v252: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T08:59:53.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: audit 2021-11-20T08:59:52.413004+0000 mon.smithi093 (mon.0) 741 : audit [DBG] from='client.? 172.21.15.93:0/1857708751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:53.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: 2021-11-20T08:59:53.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: cluster 2021-11-20T08:59:52 2021-11-20T08:59:53.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: .600148+0000 mon.smithi093 (mon.0) 742 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:reconnect 2021-11-20T08:59:53.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: cluster 2021-11-20T08:59:53.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: 2021-11-20T08:59:52. 2021-11-20T08:59:53.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: 600200+0000 mon.smithi093 (mon.0) 743 : cluster 2021-11-20T08:59:53.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:52 smithi093 conmon[33522]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:reconnect} 2 up:standby 2021-11-20T08:59:53.426 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:53.555 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 -- 172.21.15.93:0/969400562 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f6a60 msgr2=0x7fd5340f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:53.556 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 --2- 172.21.15.93:0/969400562 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f6a60 0x7fd5340f6e80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fd528004660 tx=0x7fd528009b30).stop 2021-11-20T08:59:53.556 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 -- 172.21.15.93:0/969400562 shutdown_connections 2021-11-20T08:59:53.556 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 --2- 172.21.15.93:0/969400562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5340f7400 0x7fd5340f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:53.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 --2- 172.21.15.93:0/969400562 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f6a60 0x7fd5340f6e80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:53.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 -- 172.21.15.93:0/969400562 >> 172.21.15.93:0/969400562 conn(0x7fd5340f0360 msgr2=0x7fd5340f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:53.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 -- 172.21.15.93:0/969400562 shutdown_connections 2021-11-20T08:59:53.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.554+0000 7fd539b48700 1 -- 172.21.15.93:0/969400562 wait complete. 2021-11-20T08:59:53.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd539b48700 1 Processor -- start 2021-11-20T08:59:53.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd539b48700 1 -- start start 2021-11-20T08:59:53.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd539b48700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5340f6a60 0x7fd534105cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:53.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd539b48700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f7400 0x7fd5341061b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:53.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd539b48700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd534106760 con 0x7fd5340f7400 2021-11-20T08:59:53.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd539b48700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd534106f60 con 0x7fd5340f6a60 2021-11-20T08:59:53.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd533fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f7400 0x7fd5341061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:53.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd533fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f7400 0x7fd5341061b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44016/0 (socket says 172.21.15.93:44016) 2021-11-20T08:59:53.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd533fff700 1 -- 172.21.15.93:0/3167206896 learned_addr learned my addr 172.21.15.93:0/3167206896 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:53.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.555+0000 7fd538b46700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5340f6a60 0x7fd534105cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:53.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd533fff700 1 -- 172.21.15.93:0/3167206896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5340f6a60 msgr2=0x7fd534105cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:53.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd533fff700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5340f6a60 0x7fd534105cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:53.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd533fff700 1 -- 172.21.15.93:0/3167206896 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd528005040 con 0x7fd5340f7400 2021-11-20T08:59:53.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd533fff700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f7400 0x7fd5341061b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fd52000e7c0 tx=0x7fd52000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:53.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd5317fa700 1 -- 172.21.15.93:0/3167206896 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5200092e0 con 0x7fd5340f7400 2021-11-20T08:59:53.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd534107240 con 0x7fd5340f7400 2021-11-20T08:59:53.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd5317fa700 1 -- 172.21.15.93:0/3167206896 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd52000f040 con 0x7fd5340f7400 2021-11-20T08:59:53.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd5317fa700 1 -- 172.21.15.93:0/3167206896 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd520014730 con 0x7fd5340f7400 2021-11-20T08:59:53.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.556+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd534107800 con 0x7fd5340f7400 2021-11-20T08:59:53.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.557+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5340f3ff0 con 0x7fd5340f7400 2021-11-20T08:59:53.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.557+0000 7fd5317fa700 1 -- 172.21.15.93:0/3167206896 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd5200077b0 con 0x7fd5340f7400 2021-11-20T08:59:53.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.558+0000 7fd5317fa700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd52405c300 0x7fd52405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:53.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.558+0000 7fd5317fa700 1 -- 172.21.15.93:0/3167206896 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd520076570 con 0x7fd5340f7400 2021-11-20T08:59:53.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.558+0000 7fd538b46700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd52405c300 0x7fd52405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:53.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.558+0000 7fd538b46700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd52405c300 0x7fd52405e7c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd528013530 tx=0x7fd528016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:53.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.561+0000 7fd5317fa700 1 -- 172.21.15.93:0/3167206896 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd520048e80 con 0x7fd5340f7400 2021-11-20T08:59:53.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.749+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd534045ad0 con 0x7fd5340f7400 2021-11-20T08:59:53.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.750+0000 7fd5317fa700 1 -- 172.21.15.93:0/3167206896 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2560 (secure 0 0 0) 0x7fd52004c4a0 con 0x7fd5340f7400 2021-11-20T08:59:53.750 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:53.751 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:53.597758+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:53.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.751+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd52405c300 msgr2=0x7fd52405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:53.753 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.751+0000 7fd539b48700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd52405c300 0x7fd52405e7c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd528013530 tx=0x7fd528016040).stop 2021-11-20T08:59:53.754 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.751+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f7400 msgr2=0x7fd5341061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:53.754 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.751+0000 7fd539b48700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f7400 0x7fd5341061b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fd52000e7c0 tx=0x7fd52000b040).stop 2021-11-20T08:59:53.754 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.752+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 shutdown_connections 2021-11-20T08:59:53.755 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.752+0000 7fd539b48700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd52405c300 0x7fd52405e7c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:53.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.752+0000 7fd539b48700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5340f6a60 0x7fd534105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:53.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.752+0000 7fd539b48700 1 --2- 172.21.15.93:0/3167206896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5340f7400 0x7fd5341061b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:53.756 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.752+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 >> 172.21.15.93:0/3167206896 conn(0x7fd5340f0360 msgr2=0x7fd5340fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:53.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.752+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 shutdown_connections 2021-11-20T08:59:53.757 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:53.752+0000 7fd539b48700 1 -- 172.21.15.93:0/3167206896 wait complete. 2021-11-20T08:59:53.758 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 68 2021-11-20T08:59:53.762 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:53.597758+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:53.763 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:rejoin 2021-11-20T08:59:54.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:53 smithi102 conmon[36472]: cluster 2021-11-20T08:59:52.619153+0000 mds.cephfs.smithi093.ljzugc (mds.0 2021-11-20T08:59:54.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:53 smithi102 conmon[36472]: ) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209995 2021-11-20T08:59:54.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:53 smithi102 conmon[36472]: cluster 2021-11-20T08:59:53.600833+0000 mon.smithi093 (mon.0) 744 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:rejoin 2021-11-20T08:59:54.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:53 smithi102 conmon[36472]: cluster 2021-11-20T08:59:53.600896+0000 mon.smithi093 (mon.0) 745 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 2 up:standby 2021-11-20T08:59:54.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:53 smithi093 conmon[33522]: cluster 2021-11-20T08:59:52.619153+0000 mds.cephfs.smithi093.ljzugc (mds.0) 2021-11-20T08:59:54.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:53 smithi093 conmon[33522]: 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209995 2021-11-20T08:59:54.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:53 smithi093 conmon[33522]: cluster 2021-11-20T08:59:53.600833+0000 mon.smithi093 (mon.0) 744 : cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:rejoin 2021-11-20T08:59:54.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:53 smithi093 conmon[33522]: cluster 2021-11-20T08:59:53.600896+0000 mon.smithi093 ( 2021-11-20T08:59:54.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:53 smithi093 conmon[33522]: mon.0) 745 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:rejoin} 2 up:standby 2021-11-20T08:59:54.764 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:54.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 -- 172.21.15.93:0/4190093551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 msgr2=0x7f40440f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:54.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 --2- 172.21.15.93:0/4190093551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f40440f6ca0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4030004660 tx=0x7f4030009b30).stop 2021-11-20T08:59:54.886 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 -- 172.21.15.93:0/4190093551 shutdown_connections 2021-11-20T08:59:54.886 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 --2- 172.21.15.93:0/4190093551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f40440f6ca0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:54.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 --2- 172.21.15.93:0/4190093551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40440f55a0 0x7f40440f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:54.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 -- 172.21.15.93:0/4190093551 >> 172.21.15.93:0/4190093551 conn(0x7f40440f0ce0 msgr2=0x7f40440f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:54.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 -- 172.21.15.93:0/4190093551 shutdown_connections 2021-11-20T08:59:54.888 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.879+0000 7f404b156700 1 -- 172.21.15.93:0/4190093551 wait complete. 2021-11-20T08:59:54.888 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f404b156700 1 Processor -- start 2021-11-20T08:59:54.889 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f404b156700 1 -- start start 2021-11-20T08:59:54.889 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f404b156700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40440f55a0 0x7f404405ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:54.889 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f404b156700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f404410bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:54.890 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f404b156700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404410c1b0 con 0x7f40440f6840 2021-11-20T08:59:54.890 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f404b156700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404410c2f0 con 0x7f40440f55a0 2021-11-20T08:59:54.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f4043fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f404410bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:54.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f4043fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f404410bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44030/0 (socket says 172.21.15.93:44030) 2021-11-20T08:59:54.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f4043fff700 1 -- 172.21.15.93:0/1161790235 learned_addr learned my addr 172.21.15.93:0/1161790235 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:54.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f4043fff700 1 -- 172.21.15.93:0/1161790235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40440f55a0 msgr2=0x7f404405ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:54.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f4043fff700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40440f55a0 0x7f404405ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:54.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f4043fff700 1 -- 172.21.15.93:0/1161790235 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030005040 con 0x7f40440f6840 2021-11-20T08:59:54.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f4043fff700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f404410bcc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f4030009830 tx=0x7f4030009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:54.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.880+0000 7f40417fa700 1 -- 172.21.15.93:0/1161790235 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4030018070 con 0x7f40440f6840 2021-11-20T08:59:54.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.881+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404410c570 con 0x7f40440f6840 2021-11-20T08:59:54.894 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.881+0000 7f40417fa700 1 -- 172.21.15.93:0/1161790235 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4030013410 con 0x7f40440f6840 2021-11-20T08:59:54.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.881+0000 7f40417fa700 1 -- 172.21.15.93:0/1161790235 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f403001c620 con 0x7f40440f6840 2021-11-20T08:59:54.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.881+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404410cad0 con 0x7f40440f6840 2021-11-20T08:59:54.895 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.881+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4044027a00 con 0x7f40440f6840 2021-11-20T08:59:54.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.882+0000 7f40417fa700 1 -- 172.21.15.93:0/1161790235 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f403001c8e0 con 0x7f40440f6840 2021-11-20T08:59:54.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.882+0000 7f40417fa700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f403405c2a0 0x7f403405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:54.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.882+0000 7f40417fa700 1 -- 172.21.15.93:0/1161790235 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4030078150 con 0x7f40440f6840 2021-11-20T08:59:54.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.882+0000 7f4048ef2700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f403405c2a0 0x7f403405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:54.897 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.883+0000 7f4048ef2700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f403405c2a0 0x7f403405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f40440f66a0 tx=0x7f4038006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:54.898 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:54.886+0000 7f40417fa700 1 -- 172.21.15.93:0/1161790235 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f403004aa00 con 0x7f40440f6840 2021-11-20T08:59:55.071 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.070+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40440fddd0 con 0x7f40440f6840 2021-11-20T08:59:55.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.071+0000 7f40417fa700 1 -- 172.21.15.93:0/1161790235 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f403004e020 con 0x7f40440f6840 2021-11-20T08:59:55.127 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:55.128 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:55.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f403405c2a0 msgr2=0x7f403405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:55.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f403405c2a0 0x7f403405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f40440f66a0 tx=0x7f4038006ab0).stop 2021-11-20T08:59:55.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 msgr2=0x7f404410bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:55.132 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f404410bcc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f4030009830 tx=0x7f4030009b30).stop 2021-11-20T08:59:55.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 shutdown_connections 2021-11-20T08:59:55.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f403405c2a0 0x7f403405e760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:55.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40440f55a0 0x7f404405ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:55.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 --2- 172.21.15.93:0/1161790235 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40440f6840 0x7f404410bcc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:55.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 >> 172.21.15.93:0/1161790235 conn(0x7f40440f0ce0 msgr2=0x7f40440fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:55.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 shutdown_connections 2021-11-20T08:59:55.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:55.072+0000 7f404b156700 1 -- 172.21.15.93:0/1161790235 wait complete. 2021-11-20T08:59:55.141 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T08:59:55.145 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: cluster 2021-11-20T08:59:53.567490+0000 2021-11-20T08:59:55.146 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 280 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T08:59:55.146 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:59:55.146 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: 11-20T08:59:53.750881+0000 mon.smithi093 (mon.0) 746 : audit [DBG] 2021-11-20T08:59:55.147 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: from='client.? 172.21.15.93:0/3167206896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:55.147 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: cluster 2021-11-20T08:59: 2021-11-20T08:59:55.147 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: 54.602157+0000 mon.smithi093 (mon.0) 747 : cluster 2021-11-20T08:59:55.148 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:clientreplay 2021-11-20T08:59:55.148 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T08:59:55.149 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: 20T08:59:54.602215+0000 mon.smithi093 (mon 2021-11-20T08:59:55.149 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:54 smithi093 conmon[33522]: .0) 748 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 2 up:standby 2021-11-20T08:59:55.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:55.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T08:59:55.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:54 smithi102 conmon[36472]: cluster 2021-11-20T08:59:53.567490+0000 mgr.smithi093.qdikkm (mgr.14162) 280 : 2021-11-20T08:59:55.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:54 smithi102 conmon[36472]: cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T08:59:55.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:54 smithi102 conmon[36472]: audit 2021-11-20T08:59:53.750881+0000 mon.smithi093 (mon.0) 746 : audit [DBG] from='client.? 172.21.15.93:0/3167206896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:55.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:54 smithi102 conmon[36472]: cluster 2021-11-20T08:59:54.602157+0000 mon.smithi093 (mon.0) 747 : 2021-11-20T08:59:55.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:54 smithi102 conmon[36472]: cluster [DBG] mds.? [v2:172.21.15.93:6826/3869812955,v1:172.21.15.93:6827/3869812955] up:clientreplay 2021-11-20T08:59:55.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:54 smithi102 conmon[36472]: cluster 2021-11-20T08:59:54.602215+0000 mon.smithi093 (mon.0) 748 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi093.ljzugc=up:clientreplay} 2 up:standby 2021-11-20T08:59:56.154 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:56.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:55 smithi102 conmon[36472]: audit 2021-11-20T08:59:55.071819+0000 mon.smithi093 (mon.0) 749 : 2021-11-20T08:59:56.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:55 smithi102 conmon[36472]: audit [DBG] from='client.? 172.21.15.93:0/1161790235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:56.181 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:55 smithi093 conmon[33522]: audit 2021-11-20T08:59:55.071819+0000 mon.smithi093 (mon. 2021-11-20T08:59:56.182 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:55 smithi093 conmon[33522]: 0) 749 : audit [DBG] from='client.? 172.21.15.93:0/1161790235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:56.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 -- 172.21.15.93:0/2134545979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 msgr2=0x7f29f80f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:56.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 --2- 172.21.15.93:0/2134545979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f80f69f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f29ec004660 tx=0x7f29ec009b30).stop 2021-11-20T08:59:56.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 -- 172.21.15.93:0/2134545979 shutdown_connections 2021-11-20T08:59:56.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 --2- 172.21.15.93:0/2134545979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29f80f7870 0x7f29f80f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 --2- 172.21.15.93:0/2134545979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f80f69f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 -- 172.21.15.93:0/2134545979 >> 172.21.15.93:0/2134545979 conn(0x7f29f80f1d10 msgr2=0x7f29f80f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 -- 172.21.15.93:0/2134545979 shutdown_connections 2021-11-20T08:59:56.294 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.289+0000 7f29fda5b700 1 -- 172.21.15.93:0/2134545979 wait complete. 2021-11-20T08:59:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fda5b700 1 Processor -- start 2021-11-20T08:59:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fda5b700 1 -- start start 2021-11-20T08:59:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fda5b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:56.295 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fda5b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29f80f7870 0x7f29f8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fda5b700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f810aa80 con 0x7f29f80f65d0 2021-11-20T08:59:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fda5b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f810abc0 con 0x7f29f80f7870 2021-11-20T08:59:56.296 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fca59700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fca59700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44040/0 (socket says 172.21.15.93:44040) 2021-11-20T08:59:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29fca59700 1 -- 172.21.15.93:0/3487369827 learned_addr learned my addr 172.21.15.93:0/3487369827 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.290+0000 7f29f7fff700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29f80f7870 0x7f29f8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:56.297 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29fca59700 1 -- 172.21.15.93:0/3487369827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29f80f7870 msgr2=0x7f29f8069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29fca59700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29f80f7870 0x7f29f8069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29fca59700 1 -- 172.21.15.93:0/3487369827 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec005040 con 0x7f29f80f65d0 2021-11-20T08:59:56.298 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29fca59700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f806b9a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f29ec009f50 tx=0x7f29ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:56.299 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29f57fa700 1 -- 172.21.15.93:0/3487369827 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29ec018070 con 0x7f29f80f65d0 2021-11-20T08:59:56.299 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29f57fa700 1 -- 172.21.15.93:0/3487369827 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29ec013410 con 0x7f29f80f65d0 2021-11-20T08:59:56.299 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29f57fa700 1 -- 172.21.15.93:0/3487369827 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29ec01c620 con 0x7f29f80f65d0 2021-11-20T08:59:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f806a400 con 0x7f29f80f65d0 2021-11-20T08:59:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.291+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f806a960 con 0x7f29f80f65d0 2021-11-20T08:59:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.292+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f8045ad0 con 0x7f29f80f65d0 2021-11-20T08:59:56.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.292+0000 7f29f57fa700 1 -- 172.21.15.93:0/3487369827 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f29ec01d420 con 0x7f29f80f65d0 2021-11-20T08:59:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.293+0000 7f29f57fa700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29e805c300 0x7f29e805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.293+0000 7f29f57fa700 1 -- 172.21.15.93:0/3487369827 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f29ec078130 con 0x7f29f80f65d0 2021-11-20T08:59:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.293+0000 7f29f7fff700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29e805c300 0x7f29e805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:56.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.293+0000 7f29f7fff700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29e805c300 0x7f29e805e7c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f29e400b830 tx=0x7f29e4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:56.303 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.297+0000 7f29f57fa700 1 -- 172.21.15.93:0/3487369827 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29ec04aa40 con 0x7f29f80f65d0 2021-11-20T08:59:56.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.486+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29f8027b40 con 0x7f29f80f65d0 2021-11-20T08:59:56.485 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.486+0000 7f29f57fa700 1 -- 172.21.15.93:0/3487369827 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f29f8027b40 con 0x7f29f80f65d0 2021-11-20T08:59:56.486 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:56.486 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:56.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29e805c300 msgr2=0x7f29e805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:56.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29e805c300 0x7f29e805e7c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f29e400b830 tx=0x7f29e4008040).stop 2021-11-20T08:59:56.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 msgr2=0x7f29f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:56.488 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f806b9a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f29ec009f50 tx=0x7f29ec009b30).stop 2021-11-20T08:59:56.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 shutdown_connections 2021-11-20T08:59:56.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29e805c300 0x7f29e805e7c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:56.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f29f80f65d0 0x7f29f806b9a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:56.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 --2- 172.21.15.93:0/3487369827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29f80f7870 0x7f29f8069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:56.489 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 >> 172.21.15.93:0/3487369827 conn(0x7f29f80f1d10 msgr2=0x7f29f80fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:56.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 shutdown_connections 2021-11-20T08:59:56.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:56.488+0000 7f29fda5b700 1 -- 172.21.15.93:0/3487369827 wait complete. 2021-11-20T08:59:56.490 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T08:59:56.499 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:56.499 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T08:59:56.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:56 smithi093 conmon[33522]: cluster 2021-11-20T08:59:55.567927+0000 mgr.smithi093.qdikkm (mgr.14162) 281 : cluster 2021-11-20T08:59:56.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:56 smithi093 conmon[33522]: [DBG] pgmap v255: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:59:56.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:56 smithi093 conmon[33522]: audit 2021-11-20T08:59:56.487538+0000 mon.smithi093 (mon.0) 750 : audit [DBG] from='client.? 172.21.15.93:0/3487369827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:56.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:56 smithi093 conmon[33522]: audit 2021 2021-11-20T08:59:56.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:56 smithi093 conmon[33522]: -11-20T08:59:56.624330+0000 mon.smithi093 (mon.0) 751 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:59:56.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:56 smithi093 conmon[33522]: audit 2021- 2021-11-20T08:59:56.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:56 smithi093 conmon[33522]: 11-20T08:59:56.625580+0000 mon.smithi093 (mon.0) 752 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:59:57.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:56 smithi102 conmon[36472]: cluster 2021-11-20T08:59:55.567927+0000 mgr.smithi093.qdikkm (mgr.14162) 281 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 3.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T08:59:57.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:56 smithi102 conmon[36472]: audit 2021-11-20T08:59:56.487538+0000 mon.smithi093 (mon.0) 750 : audit [DBG] from='client.? 172.21.15.93:0/3487369827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:57.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:56 smithi102 conmon[36472]: audit 2021-11-20T08:59:56.624330+0000 mon.smithi093 (mon.0) 751 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T08:59:57.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:56 smithi102 conmon[36472]: audit 2021-11-20T08:59:56.625580+0000 mon.smithi093 (mon.0) 752 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T08:59:57.501 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:57.631 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 -- 172.21.15.93:0/1809704933 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f5580 msgr2=0x7f41340f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:57.632 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 --2- 172.21.15.93:0/1809704933 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f5580 0x7f41340f59a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f4130004660 tx=0x7f4130009b30).stop 2021-11-20T08:59:57.632 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 -- 172.21.15.93:0/1809704933 shutdown_connections 2021-11-20T08:59:57.632 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 --2- 172.21.15.93:0/1809704933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41340f6820 0x7f41340f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:57.632 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 --2- 172.21.15.93:0/1809704933 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f5580 0x7f41340f59a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:57.633 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 -- 172.21.15.93:0/1809704933 >> 172.21.15.93:0/1809704933 conn(0x7f41340f0ce0 msgr2=0x7f41340f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:57.633 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 -- 172.21.15.93:0/1809704933 shutdown_connections 2021-11-20T08:59:57.633 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 -- 172.21.15.93:0/1809704933 wait complete. 2021-11-20T08:59:57.633 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 Processor -- start 2021-11-20T08:59:57.633 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.629+0000 7f413b596700 1 -- start start 2021-11-20T08:59:57.634 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f413b596700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f6820 0x7f41340604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:57.634 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f413b596700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4134109e30 0x7f413410c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:57.634 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f413b596700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413410c710 con 0x7f41340f6820 2021-11-20T08:59:57.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f413b596700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413410c850 con 0x7f4134109e30 2021-11-20T08:59:57.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4139332700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f6820 0x7f41340604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:57.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4139332700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f6820 0x7f41340604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44050/0 (socket says 172.21.15.93:44050) 2021-11-20T08:59:57.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4139332700 1 -- 172.21.15.93:0/3919469446 learned_addr learned my addr 172.21.15.93:0/3919469446 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:57.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4139332700 1 -- 172.21.15.93:0/3919469446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4134109e30 msgr2=0x7f413410c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:57.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4139332700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4134109e30 0x7f413410c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:57.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4139332700 1 -- 172.21.15.93:0/3919469446 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4130005040 con 0x7f41340f6820 2021-11-20T08:59:57.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4139332700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f6820 0x7f41340604d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f4130009830 tx=0x7f4130009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:57.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4129ffb700 1 -- 172.21.15.93:0/3919469446 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4130019070 con 0x7f41340f6820 2021-11-20T08:59:57.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4129ffb700 1 -- 172.21.15.93:0/3919469446 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4130013410 con 0x7f41340f6820 2021-11-20T08:59:57.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413410ca50 con 0x7f41340f6820 2021-11-20T08:59:57.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f4129ffb700 1 -- 172.21.15.93:0/3919469446 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4130014530 con 0x7f41340f6820 2021-11-20T08:59:57.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.630+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413410cf20 con 0x7f41340f6820 2021-11-20T08:59:57.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.631+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4134027a00 con 0x7f41340f6820 2021-11-20T08:59:57.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.632+0000 7f4129ffb700 1 -- 172.21.15.93:0/3919469446 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4130014690 con 0x7f41340f6820 2021-11-20T08:59:57.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.632+0000 7f4129ffb700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f412005c2a0 0x7f412005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:57.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.632+0000 7f4129ffb700 1 -- 172.21.15.93:0/3919469446 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4130077650 con 0x7f41340f6820 2021-11-20T08:59:57.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.632+0000 7f4138b31700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f412005c2a0 0x7f412005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:57.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.632+0000 7f4138b31700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f412005c2a0 0x7f412005e760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f4124004a10 tx=0x7f41240099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:57.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.636+0000 7f4129ffb700 1 -- 172.21.15.93:0/3919469446 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4130049f00 con 0x7f41340f6820 2021-11-20T08:59:57.820 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.820+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41340fdd10 con 0x7f41340f6820 2021-11-20T08:59:57.820 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.820+0000 7f4129ffb700 1 -- 172.21.15.93:0/3919469446 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f413004d520 con 0x7f41340f6820 2021-11-20T08:59:57.821 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:57.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f412005c2a0 msgr2=0x7f412005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:57.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f412005c2a0 0x7f412005e760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f4124004a10 tx=0x7f41240099e0).stop 2021-11-20T08:59:57.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f6820 msgr2=0x7f41340604d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:57.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f6820 0x7f41340604d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f4130009830 tx=0x7f4130009b30).stop 2021-11-20T08:59:57.824 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 shutdown_connections 2021-11-20T08:59:57.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f412005c2a0 0x7f412005e760 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:57.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41340f6820 0x7f41340604d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:57.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 --2- 172.21.15.93:0/3919469446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4134109e30 0x7f413410c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T08:59:57.825 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 >> 172.21.15.93:0/3919469446 conn(0x7f41340f0ce0 msgr2=0x7f41340fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:57.826 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 shutdown_connections 2021-11-20T08:59:57.826 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:57.822+0000 7f413b596700 1 -- 172.21.15.93:0/3919469446 wait complete. 2021-11-20T08:59:57.826 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T08:59:57.832 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:57.833 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T08:59:58.835 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T08:59:58.970 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 -- 172.21.15.93:0/927889859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 msgr2=0x7fd9600f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:58.970 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 --2- 172.21.15.93:0/927889859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd9600f6ca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fd954004660 tx=0x7fd954009b30).stop 2021-11-20T08:59:58.970 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 -- 172.21.15.93:0/927889859 shutdown_connections 2021-11-20T08:59:58.971 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 --2- 172.21.15.93:0/927889859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd9600f6ca0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:58.971 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 --2- 172.21.15.93:0/927889859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9600f55a0 0x7fd9600f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:58.971 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 -- 172.21.15.93:0/927889859 >> 172.21.15.93:0/927889859 conn(0x7fd9600f0ce0 msgr2=0x7fd9600f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:58.971 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 -- 172.21.15.93:0/927889859 shutdown_connections 2021-11-20T08:59:58.972 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.969+0000 7fd96518d700 1 -- 172.21.15.93:0/927889859 wait complete. 2021-11-20T08:59:58.972 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd96518d700 1 Processor -- start 2021-11-20T08:59:58.972 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd96518d700 1 -- start start 2021-11-20T08:59:58.972 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd96518d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9600f55a0 0x7fd96005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:58.973 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd96518d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd96010bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:58.973 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd96518d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96010c110 con 0x7fd9600f6840 2021-11-20T08:59:58.973 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd96518d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96010c250 con 0x7fd9600f55a0 2021-11-20T08:59:58.973 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd96010bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:58.974 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd96010bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44064/0 (socket says 172.21.15.93:44064) 2021-11-20T08:59:58.974 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95e59c700 1 -- 172.21.15.93:0/248235443 learned_addr learned my addr 172.21.15.93:0/248235443 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T08:59:58.974 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95ed9d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9600f55a0 0x7fd96005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:58.974 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95e59c700 1 -- 172.21.15.93:0/248235443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9600f55a0 msgr2=0x7fd96005ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:58.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95e59c700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9600f55a0 0x7fd96005ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:58.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95e59c700 1 -- 172.21.15.93:0/248235443 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd954005040 con 0x7fd9600f6840 2021-11-20T08:59:58.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd95e59c700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd96010bc20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fd954000c00 tx=0x7fd95400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:58.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.970+0000 7fd9477fe700 1 -- 172.21.15.93:0/248235443 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd954019070 con 0x7fd9600f6840 2021-11-20T08:59:58.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.971+0000 7fd9477fe700 1 -- 172.21.15.93:0/248235443 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd954007e70 con 0x7fd9600f6840 2021-11-20T08:59:58.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.971+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96010c450 con 0x7fd9600f6840 2021-11-20T08:59:58.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.971+0000 7fd9477fe700 1 -- 172.21.15.93:0/248235443 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd954014cc0 con 0x7fd9600f6840 2021-11-20T08:59:58.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.971+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96010c9b0 con 0x7fd9600f6840 2021-11-20T08:59:58.977 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.972+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd960027a00 con 0x7fd9600f6840 2021-11-20T08:59:58.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.972+0000 7fd9477fe700 1 -- 172.21.15.93:0/248235443 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd954025070 con 0x7fd9600f6840 2021-11-20T08:59:58.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.972+0000 7fd9477fe700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd94805c2a0 0x7fd94805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T08:59:58.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.972+0000 7fd9477fe700 1 -- 172.21.15.93:0/248235443 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd9540772b0 con 0x7fd9600f6840 2021-11-20T08:59:58.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.972+0000 7fd95ed9d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd94805c2a0 0x7fd94805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T08:59:58.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.973+0000 7fd95ed9d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd94805c2a0 0x7fd94805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd9600f66a0 tx=0x7fd95000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T08:59:58.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:58.976+0000 7fd9477fe700 1 -- 172.21.15.93:0/248235443 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd954049bc0 con 0x7fd9600f6840 2021-11-20T08:59:59.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.160+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9600fdd30 con 0x7fd9600f6840 2021-11-20T08:59:59.159 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.160+0000 7fd9477fe700 1 -- 172.21.15.93:0/248235443 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7fd95404d1e0 con 0x7fd9600f6840 2021-11-20T08:59:59.160 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T08:59:59.160 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T08:59:59.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd94805c2a0 msgr2=0x7fd94805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:59.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd94805c2a0 0x7fd94805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd9600f66a0 tx=0x7fd95000b040).stop 2021-11-20T08:59:59.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 msgr2=0x7fd96010bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T08:59:59.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd96010bc20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fd954000c00 tx=0x7fd95400b040).stop 2021-11-20T08:59:59.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 shutdown_connections 2021-11-20T08:59:59.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd94805c2a0 0x7fd94805e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:59.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9600f55a0 0x7fd96005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:59.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 --2- 172.21.15.93:0/248235443 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9600f6840 0x7fd96010bc20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T08:59:59.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 >> 172.21.15.93:0/248235443 conn(0x7fd9600f0ce0 msgr2=0x7fd9600fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T08:59:59.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 shutdown_connections 2021-11-20T08:59:59.164 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T08:59:59.162+0000 7fd96518d700 1 -- 172.21.15.93:0/248235443 wait complete. 2021-11-20T08:59:59.164 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T08:59:59.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:58 smithi102 conmon[36472]: cluster 2021-11-20T08:59:57.568340+0000 mgr.smithi093.qdikkm (mgr.14162) 282 : cluster [DBG] 2021-11-20T08:59:59.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:58 smithi102 conmon[36472]: pgmap v256: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:59.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:58 smithi102 conmon[36472]: audit 2021-11-20T08:59:57.821227+0000 mon.smithi093 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.93:0/3919469446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:59.171 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:58 smithi093 conmon[33522]: cluster 2021-11-20T08:59:57.568340+0000 mgr.smithi093.qdikkm (mgr.14162) 282 : cluster 2021-11-20T08:59:59.171 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:58 smithi093 conmon[33522]: [DBG] pgmap v256: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T08:59:59.172 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:58 smithi093 conmon[33522]: audit 2021-11-20T08:59:57.821227+0000 mon.smithi093 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.93:0/3919469446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T08:59:59.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T08:59:59.174 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T08:59:59.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 08:59:59 smithi093 conmon[33522]: audit 2021-11-20T08:59:59.161100+0000 mon.smithi093 (mon.0) 754 : audit [DBG] from='client.? 172.21.15.93:0/248235443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:00.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:59 smithi102 conmon[36472]: audit 2021-11-20T08:59:59.161100+0000 mon.smithi093 (mon.0) 754 : audit 2021-11-20T09:00:00.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 08:59:59 smithi102 conmon[36472]: [DBG] from='client.? 172.21.15.93:0/248235443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:00.175 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:00.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 -- 172.21.15.93:0/1777273440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 msgr2=0x7f66400f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:00.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 --2- 172.21.15.93:0/1777273440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f66400f7da0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f6630004660 tx=0x7f6630009b30).stop 2021-11-20T09:00:00.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 -- 172.21.15.93:0/1777273440 shutdown_connections 2021-11-20T09:00:00.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 --2- 172.21.15.93:0/1777273440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f66400f7da0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:00.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 --2- 172.21.15.93:0/1777273440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66400f66a0 0x7f66400f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:00.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 -- 172.21.15.93:0/1777273440 >> 172.21.15.93:0/1777273440 conn(0x7f66400f1e00 msgr2=0x7f66400f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:00.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 -- 172.21.15.93:0/1777273440 shutdown_connections 2021-11-20T09:00:00.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.306+0000 7f664787e700 1 -- 172.21.15.93:0/1777273440 wait complete. 2021-11-20T09:00:00.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f664787e700 1 Processor -- start 2021-11-20T09:00:00.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f664787e700 1 -- start start 2021-11-20T09:00:00.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f664787e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66400f66a0 0x7f6640106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:00.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f664787e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f6640107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:00.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f664787e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640107900 con 0x7f66400f7940 2021-11-20T09:00:00.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f664787e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640107a40 con 0x7f66400f66a0 2021-11-20T09:00:00.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f6644e19700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f6640107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:00.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f6644e19700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f6640107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44074/0 (socket says 172.21.15.93:44074) 2021-11-20T09:00:00.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f6644e19700 1 -- 172.21.15.93:0/4231912451 learned_addr learned my addr 172.21.15.93:0/4231912451 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:00.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f664561a700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66400f66a0 0x7f6640106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:00.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f6644e19700 1 -- 172.21.15.93:0/4231912451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66400f66a0 msgr2=0x7f6640106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:00.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f6644e19700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66400f66a0 0x7f6640106e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:00.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f6644e19700 1 -- 172.21.15.93:0/4231912451 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630005040 con 0x7f66400f7940 2021-11-20T09:00:00.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.307+0000 7f6644e19700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f6640107350 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f6630009f50 tx=0x7f6630009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:00.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.308+0000 7f6635ffb700 1 -- 172.21.15.93:0/4231912451 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6630018070 con 0x7f66400f7940 2021-11-20T09:00:00.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.308+0000 7f6635ffb700 1 -- 172.21.15.93:0/4231912451 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6630013410 con 0x7f66400f7940 2021-11-20T09:00:00.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.308+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664010c480 con 0x7f66400f7940 2021-11-20T09:00:00.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.308+0000 7f6635ffb700 1 -- 172.21.15.93:0/4231912451 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663001c3c0 con 0x7f66400f7940 2021-11-20T09:00:00.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.308+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664010ca40 con 0x7f66400f7940 2021-11-20T09:00:00.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.309+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6640027a00 con 0x7f66400f7940 2021-11-20T09:00:00.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.309+0000 7f6635ffb700 1 -- 172.21.15.93:0/4231912451 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f663001d420 con 0x7f66400f7940 2021-11-20T09:00:00.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.310+0000 7f6635ffb700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f662c05c2a0 0x7f662c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:00.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.310+0000 7f6635ffb700 1 -- 172.21.15.93:0/4231912451 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6630077ef0 con 0x7f66400f7940 2021-11-20T09:00:00.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.310+0000 7f664561a700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f662c05c2a0 0x7f662c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:00.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.310+0000 7f664561a700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f662c05c2a0 0x7f662c05e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f66400f77a0 tx=0x7f663c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:00.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.313+0000 7f6635ffb700 1 -- 172.21.15.93:0/4231912451 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f663004a800 con 0x7f66400f7940 2021-11-20T09:00:00.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.499+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664010c610 con 0x7f66400f7940 2021-11-20T09:00:00.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.499+0000 7f6635ffb700 1 -- 172.21.15.93:0/4231912451 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f664010c610 con 0x7f66400f7940 2021-11-20T09:00:00.500 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:00.501 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:00.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f662c05c2a0 msgr2=0x7f662c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:00.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f662c05c2a0 0x7f662c05e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f66400f77a0 tx=0x7f663c00b040).stop 2021-11-20T09:00:00.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 msgr2=0x7f6640107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:00.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f6640107350 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f6630009f50 tx=0x7f6630009b30).stop 2021-11-20T09:00:00.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 shutdown_connections 2021-11-20T09:00:00.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f662c05c2a0 0x7f662c05e760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:00.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66400f66a0 0x7f6640106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:00.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 --2- 172.21.15.93:0/4231912451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66400f7940 0x7f6640107350 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:00.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 >> 172.21.15.93:0/4231912451 conn(0x7f66400f1e00 msgr2=0x7f66400fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:00.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 shutdown_connections 2021-11-20T09:00:00.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:00.501+0000 7f664787e700 1 -- 172.21.15.93:0/4231912451 wait complete. 2021-11-20T09:00:00.505 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:00.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:00.515 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:00.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:00 smithi093 conmon[33522]: cluster 2021-11-20T08:59:59.568975+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T09:00:00.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:00 smithi093 conmon[33522]: 14162) 283 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:00.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:00 smithi093 conmon[33522]: cluster 2021-11-20T09:00:00.000179+0000 mon.smithi093 (mon.0) 755 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem is degraded 2021-11-20T09:00:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:00 smithi093 conmon[33522]: cluster 2021-11-20T09:00:00.000229+0000 mon.smithi093 (mon.0) 756 : cluster [WRN] [WRN] FS_DEGRADED: 1 filesystem is degraded 2021-11-20T09:00:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:00 smithi093 conmon[33522]: cluster 2021-11-20T09:00:00.000269+0000 mon.smithi093 (mon.0) 757 : cluster [WRN] fs cephfs is degraded 2021-11-20T09:00:00.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:00 smithi093 conmon[33522]: audit 2021-11-20T09:00:00.500510+0000 mon.smithi093 (mon.0) 758 : audit [DBG] from='client.? 172.21.15.93:0/4231912451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:01.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: cluster 2021-11-20T08:59:59.568975+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T09:00:01.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: .14162) 283 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:01.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: cluster 2021-11-20T09:00:00.000179+0000 mon.smithi093 (mon.0) 755 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem is degraded 2021-11-20T09:00:01.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: cluster 2021-11-20T09:00:01.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: 2021-11-20T09:00:00.000229+0000 mon.smithi093 (mon.0) 756 : cluster [WRN] [WRN] FS_DEGRADED: 1 filesystem is degraded 2021-11-20T09:00:01.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: cluster 2021-11-20T09:00 2021-11-20T09:00:01.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: :00.000269+0000 mon.smithi093 (mon.0) 757 : cluster [WRN] fs cephfs is degraded 2021-11-20T09:00:01.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: audit 2021 2021-11-20T09:00:01.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:00 smithi102 conmon[36472]: -11-20T09:00:00.500510+0000 mon.smithi093 (mon.0) 758 : audit [DBG] from='client.? 172.21.15.93:0/4231912451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:01.516 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:01.658 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 -- 172.21.15.93:0/669459688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 msgr2=0x7f774c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:01.659 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 --2- 172.21.15.93:0/669459688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c0f59a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f77480099e0 tx=0x7f7748004d10).stop 2021-11-20T09:00:01.659 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 -- 172.21.15.93:0/669459688 shutdown_connections 2021-11-20T09:00:01.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 --2- 172.21.15.93:0/669459688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f774c0f6820 0x7f774c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:01.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 --2- 172.21.15.93:0/669459688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c0f59a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:01.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 -- 172.21.15.93:0/669459688 >> 172.21.15.93:0/669459688 conn(0x7f774c0f0ce0 msgr2=0x7f774c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:01.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 -- 172.21.15.93:0/669459688 shutdown_connections 2021-11-20T09:00:01.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 -- 172.21.15.93:0/669459688 wait complete. 2021-11-20T09:00:01.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 Processor -- start 2021-11-20T09:00:01.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.651+0000 7f77535ef700 1 -- start start 2021-11-20T09:00:01.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f77535ef700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:01.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f77535ef700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f774c0f6820 0x7f774c10bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:01.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f77535ef700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c10c2a0 con 0x7f774c0f5580 2021-11-20T09:00:01.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f77535ef700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c10c3e0 con 0x7f774c0f6820 2021-11-20T09:00:01.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f775138b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:01.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f775138b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c060530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44082/0 (socket says 172.21.15.93:44082) 2021-11-20T09:00:01.663 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f775138b700 1 -- 172.21.15.93:0/1581870333 learned_addr learned my addr 172.21.15.93:0/1581870333 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:01.663 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f7750b8a700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f774c0f6820 0x7f774c10bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:01.663 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f775138b700 1 -- 172.21.15.93:0/1581870333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f774c0f6820 msgr2=0x7f774c10bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:01.663 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f775138b700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f774c0f6820 0x7f774c10bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:01.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f775138b700 1 -- 172.21.15.93:0/1581870333 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7748009640 con 0x7f774c0f5580 2021-11-20T09:00:01.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f775138b700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c060530 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f77480078b0 tx=0x7f7748004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:01.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f7741ffb700 1 -- 172.21.15.93:0/1581870333 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f774800e030 con 0x7f774c0f5580 2021-11-20T09:00:01.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f7741ffb700 1 -- 172.21.15.93:0/1581870333 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7748007d50 con 0x7f774c0f5580 2021-11-20T09:00:01.665 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774c10c5e0 con 0x7f774c0f5580 2021-11-20T09:00:01.665 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f7741ffb700 1 -- 172.21.15.93:0/1581870333 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7748018070 con 0x7f774c0f5580 2021-11-20T09:00:01.665 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.652+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774c10ca90 con 0x7f774c0f5580 2021-11-20T09:00:01.666 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.653+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f774c046330 con 0x7f774c0f5580 2021-11-20T09:00:01.666 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.654+0000 7f7741ffb700 1 -- 172.21.15.93:0/1581870333 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f77480137e0 con 0x7f774c0f5580 2021-11-20T09:00:01.666 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.654+0000 7f7741ffb700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f773805c2a0 0x7f773805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:01.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.654+0000 7f7741ffb700 1 -- 172.21.15.93:0/1581870333 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7748077ff0 con 0x7f774c0f5580 2021-11-20T09:00:01.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.654+0000 7f7750b8a700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f773805c2a0 0x7f773805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:01.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.654+0000 7f7750b8a700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f773805c2a0 0x7f773805e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f773c004c40 tx=0x7f773c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:01.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.658+0000 7f7741ffb700 1 -- 172.21.15.93:0/1581870333 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f774804a9b0 con 0x7f774c0f5580 2021-11-20T09:00:01.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.848+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f774c027a00 con 0x7f774c0f5580 2021-11-20T09:00:01.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.848+0000 7f7741ffb700 1 -- 172.21.15.93:0/1581870333 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f774804dfd0 con 0x7f774c0f5580 2021-11-20T09:00:01.847 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:01.848 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:01.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f773805c2a0 msgr2=0x7f773805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:01.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f773805c2a0 0x7f773805e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f773c004c40 tx=0x7f773c00b040).stop 2021-11-20T09:00:01.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 msgr2=0x7f774c060530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:01.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c060530 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f77480078b0 tx=0x7f7748004d10).stop 2021-11-20T09:00:01.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 shutdown_connections 2021-11-20T09:00:01.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f773805c2a0 0x7f773805e760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:01.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f774c0f5580 0x7f774c060530 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:01.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 --2- 172.21.15.93:0/1581870333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f774c0f6820 0x7f774c10bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:01.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 >> 172.21.15.93:0/1581870333 conn(0x7f774c0f0ce0 msgr2=0x7f774c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:01.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 shutdown_connections 2021-11-20T09:00:01.851 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:01.850+0000 7f77535ef700 1 -- 172.21.15.93:0/1581870333 wait complete. 2021-11-20T09:00:01.852 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:01.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:01.862 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:02.866 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:03.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.996+0000 7f062af4a700 1 -- 172.21.15.93:0/502686841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f5810 msgr2=0x7f06240f5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:03.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.996+0000 7f062af4a700 1 --2- 172.21.15.93:0/502686841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f5810 0x7f06240f5c90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0614004660 tx=0x7f0614009b30).stop 2021-11-20T09:00:03.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.996+0000 7f062af4a700 1 -- 172.21.15.93:0/502686841 shutdown_connections 2021-11-20T09:00:03.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.996+0000 7f062af4a700 1 --2- 172.21.15.93:0/502686841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f5810 0x7f06240f5c90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:03.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.996+0000 7f062af4a700 1 --2- 172.21.15.93:0/502686841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06240f6b00 0x7f06240f6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:03.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.996+0000 7f062af4a700 1 -- 172.21.15.93:0/502686841 >> 172.21.15.93:0/502686841 conn(0x7f06240f0360 msgr2=0x7f06240f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:03.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 -- 172.21.15.93:0/502686841 shutdown_connections 2021-11-20T09:00:03.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 -- 172.21.15.93:0/502686841 wait complete. 2021-11-20T09:00:03.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 Processor -- start 2021-11-20T09:00:03.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 -- start start 2021-11-20T09:00:03.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06240f5810 0x7f0624105c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:03.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f6b00 0x7f06241060f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:03.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06241066a0 con 0x7f06240f6b00 2021-11-20T09:00:03.004 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f062af4a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06241067e0 con 0x7f06240f5810 2021-11-20T09:00:03.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f0629747700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f6b00 0x7f06241060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:03.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f0629747700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f6b00 0x7f06241060f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44096/0 (socket says 172.21.15.93:44096) 2021-11-20T09:00:03.005 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f0629747700 1 -- 172.21.15.93:0/4034120895 learned_addr learned my addr 172.21.15.93:0/4034120895 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:03.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.997+0000 7f0629f48700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06240f5810 0x7f0624105c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:03.006 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f0629747700 1 -- 172.21.15.93:0/4034120895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06240f5810 msgr2=0x7f0624105c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:03.007 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f0629747700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06240f5810 0x7f0624105c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:03.007 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f0629747700 1 -- 172.21.15.93:0/4034120895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0614005040 con 0x7f06240f6b00 2021-11-20T09:00:03.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f0629747700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f6b00 0x7f06241060f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f0614009830 tx=0x7f0614009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:03.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f061a7fc700 1 -- 172.21.15.93:0/4034120895 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0614018070 con 0x7f06240f6b00 2021-11-20T09:00:03.008 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f061a7fc700 1 -- 172.21.15.93:0/4034120895 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0614013410 con 0x7f06240f6b00 2021-11-20T09:00:03.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0624107120 con 0x7f06240f6b00 2021-11-20T09:00:03.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f061a7fc700 1 -- 172.21.15.93:0/4034120895 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f061401c620 con 0x7f06240f6b00 2021-11-20T09:00:03.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.998+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0624107680 con 0x7f06240f6b00 2021-11-20T09:00:03.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.999+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0624045ad0 con 0x7f06240f6b00 2021-11-20T09:00:03.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.999+0000 7f061a7fc700 1 -- 172.21.15.93:0/4034120895 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f061401c8e0 con 0x7f06240f6b00 2021-11-20T09:00:03.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:02.999+0000 7f061a7fc700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f061005c300 0x7f061005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:03.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.000+0000 7f061a7fc700 1 -- 172.21.15.93:0/4034120895 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0614078150 con 0x7f06240f6b00 2021-11-20T09:00:03.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.000+0000 7f0629f48700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f061005c300 0x7f061005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:03.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.000+0000 7f0629f48700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f061005c300 0x7f061005e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f0620007fd0 tx=0x7f0620009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:03.011 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.003+0000 7f061a7fc700 1 -- 172.21.15.93:0/4034120895 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f061404aa00 con 0x7f06240f6b00 2021-11-20T09:00:03.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:02 smithi102 conmon[36472]: cluster 2021-11-20T09:00:01.569441+0000 mgr.smithi093.qdikkm (mgr.14162) 284 : cluster [DBG] pgmap v258: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:03.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:02 smithi102 conmon[36472]: audit 2021-11-20T09:00:01.849105+0000 mon.smithi093 (mon.0) 759 : audit [DBG] from='client.? 172.21.15.93:0/1581870333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:03.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.186+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0624027a00 con 0x7f06240f6b00 2021-11-20T09:00:03.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.186+0000 7f061a7fc700 1 -- 172.21.15.93:0/4034120895 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f061404e020 con 0x7f06240f6b00 2021-11-20T09:00:03.186 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:03.186 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:03.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f061005c300 msgr2=0x7f061005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:03.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f061005c300 0x7f061005e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f0620007fd0 tx=0x7f0620009040).stop 2021-11-20T09:00:03.188 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f6b00 msgr2=0x7f06241060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:03.189 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f6b00 0x7f06241060f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f0614009830 tx=0x7f0614009b30).stop 2021-11-20T09:00:03.190 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 shutdown_connections 2021-11-20T09:00:03.190 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f061005c300 0x7f061005e7c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:03.190 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06240f5810 0x7f0624105c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:03.191 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 --2- 172.21.15.93:0/4034120895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06240f6b00 0x7f06241060f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:03.191 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 >> 172.21.15.93:0/4034120895 conn(0x7f06240f0360 msgr2=0x7f06240fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:03.192 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 shutdown_connections 2021-11-20T09:00:03.192 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:03.188+0000 7f062af4a700 1 -- 172.21.15.93:0/4034120895 wait complete. 2021-11-20T09:00:03.192 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:03.197 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:02 smithi093 conmon[33522]: cluster 2021-11-20T09:00:01.569441+0000 mgr.smithi093.qdikkm (mgr.14162) 284 : cluster [DBG] 2021-11-20T09:00:03.198 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:02 smithi093 conmon[33522]: pgmap v258: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:03.198 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:02 smithi093 conmon[33522]: audit 2021-11-20T09:00:01.849105+0000 mon.smithi093 (mon.0) 759 : audit [DBG] from='client.? 172.21.15.93:0/1581870333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:03.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:03.201 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:04.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:03 smithi102 conmon[36472]: audit 2021-11-20T09:00:03.187334+0000 mon.smithi093 (mon.0) 760 : audit 2021-11-20T09:00:04.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:03 smithi102 conmon[36472]: [DBG] from='client.? 172.21.15.93:0/4034120895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:04.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:03 smithi102 conmon[36472]: audit 2021-11-20T09:00:03.602622+0000 mon.smithi093 (mon.0) 761 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T09:00:04.203 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:04.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:03 smithi093 conmon[33522]: audit 2021-11-20T09:00: 2021-11-20T09:00:04.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:03 smithi093 conmon[33522]: 03.187334+0000 mon.smithi093 (mon.0) 760 : audit [DBG] from='client.? 172.21.15.93:0/4034120895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:04.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:03 smithi093 conmon[33522]: audit 2021-11-20T09:00:03.602622+0000 mon.smithi093 ( 2021-11-20T09:00:04.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:03 smithi093 conmon[33522]: mon.0) 761 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T09:00:04.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 -- 172.21.15.93:0/496473783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f55c0 msgr2=0x7f5f0c0f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 --2- 172.21.15.93:0/496473783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f55c0 0x7f5f0c0f59e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f5efc0099e0 tx=0x7f5efc004d10).stop 2021-11-20T09:00:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 -- 172.21.15.93:0/496473783 shutdown_connections 2021-11-20T09:00:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 --2- 172.21.15.93:0/496473783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0f6860 0x7f5f0c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:04.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 --2- 172.21.15.93:0/496473783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f55c0 0x7f5f0c0f59e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 -- 172.21.15.93:0/496473783 >> 172.21.15.93:0/496473783 conn(0x7f5f0c0f0d40 msgr2=0x7f5f0c0f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 -- 172.21.15.93:0/496473783 shutdown_connections 2021-11-20T09:00:04.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 -- 172.21.15.93:0/496473783 wait complete. 2021-11-20T09:00:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 Processor -- start 2021-11-20T09:00:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.320+0000 7f5f11b12700 1 -- start start 2021-11-20T09:00:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f11b12700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0f55c0 0x7f5f0c107000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f11b12700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f6860 0x7f5f0c1074f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:04.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f11b12700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c107aa0 con 0x7f5f0c0f6860 2021-11-20T09:00:04.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f11b12700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c107be0 con 0x7f5f0c0f55c0 2021-11-20T09:00:04.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f6860 0x7f5f0c1074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:04.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f6860 0x7f5f0c1074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44106/0 (socket says 172.21.15.93:44106) 2021-11-20T09:00:04.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0affd700 1 -- 172.21.15.93:0/2023429582 learned_addr learned my addr 172.21.15.93:0/2023429582 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:04.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0b7fe700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0f55c0 0x7f5f0c107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:04.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0affd700 1 -- 172.21.15.93:0/2023429582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0f55c0 msgr2=0x7f5f0c107000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:04.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0affd700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0f55c0 0x7f5f0c107000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:04.344 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0affd700 1 -- 172.21.15.93:0/2023429582 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc009640 con 0x7f5f0c0f6860 2021-11-20T09:00:04.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f0affd700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f6860 0x7f5f0c1074f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f5f0000e7f0 tx=0x7f5f0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:04.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5ef3fff700 1 -- 172.21.15.93:0/2023429582 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f000092e0 con 0x7f5f0c0f6860 2021-11-20T09:00:04.345 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5ef3fff700 1 -- 172.21.15.93:0/2023429582 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f0000f040 con 0x7f5f0c0f6860 2021-11-20T09:00:04.346 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5ef3fff700 1 -- 172.21.15.93:0/2023429582 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f00014750 con 0x7f5f0c0f6860 2021-11-20T09:00:04.348 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c10c680 con 0x7f5f0c0f6860 2021-11-20T09:00:04.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.321+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c10cc40 con 0x7f5f0c0f6860 2021-11-20T09:00:04.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.322+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c0fdd50 con 0x7f5f0c0f6860 2021-11-20T09:00:04.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.323+0000 7f5ef3fff700 1 -- 172.21.15.93:0/2023429582 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5f0001b030 con 0x7f5f0c0f6860 2021-11-20T09:00:04.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.323+0000 7f5ef3fff700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5ef405c2a0 0x7f5ef405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:04.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.323+0000 7f5ef3fff700 1 -- 172.21.15.93:0/2023429582 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5f00075f50 con 0x7f5f0c0f6860 2021-11-20T09:00:04.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.323+0000 7f5f0b7fe700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5ef405c2a0 0x7f5ef405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:04.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.323+0000 7f5f0b7fe700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5ef405c2a0 0x7f5ef405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5efc003380 tx=0x7f5efc015040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:04.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.327+0000 7f5ef3fff700 1 -- 172.21.15.93:0/2023429582 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f000a22b0 con 0x7f5f0c0f6860 2021-11-20T09:00:04.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.512+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f0c10d330 con 0x7f5f0c0f6860 2021-11-20T09:00:04.511 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.512+0000 7f5ef3fff700 1 -- 172.21.15.93:0/2023429582 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f5f00019020 con 0x7f5f0c0f6860 2021-11-20T09:00:04.512 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:04.512 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:04.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5ef405c2a0 msgr2=0x7f5ef405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:04.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5ef405c2a0 0x7f5ef405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5efc003380 tx=0x7f5efc015040).stop 2021-11-20T09:00:04.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f6860 msgr2=0x7f5f0c1074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:04.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f6860 0x7f5f0c1074f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f5f0000e7f0 tx=0x7f5f0000b040).stop 2021-11-20T09:00:04.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 shutdown_connections 2021-11-20T09:00:04.518 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f5ef405c2a0 0x7f5ef405e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:04.519 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f0c0f55c0 0x7f5f0c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:04.519 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 --2- 172.21.15.93:0/2023429582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f0c0f6860 0x7f5f0c1074f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:04.519 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 >> 172.21.15.93:0/2023429582 conn(0x7f5f0c0f0d40 msgr2=0x7f5f0c05fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:04.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 shutdown_connections 2021-11-20T09:00:04.520 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:04.514+0000 7f5f11b12700 1 -- 172.21.15.93:0/2023429582 wait complete. 2021-11-20T09:00:04.520 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:04.525 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:04.525 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:05.424 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:04 smithi102 conmon[36472]: cluster 2021-11-20T09:00:03.569824+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T09:00:05.424 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:04 smithi102 conmon[36472]: ) 285 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 171 B/s wr, 1 op/s 2021-11-20T09:00:05.424 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:04 smithi102 conmon[36472]: audit 2021-11-20T09:00:03.972308+0000 mon.smithi093 (mon.0) 762 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T09:00:05.425 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:04 smithi102 conmon[36472]: audit 2021-11-20T09:00:04.513202+0000 mon.smithi093 (mon.0) 763 : audit [DBG] from='client.? 172.21.15.93:0/2023429582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:05.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:04 smithi093 conmon[33522]: cluster 2021-11-20T09:00:03.569824+0000 2021-11-20T09:00:05.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:04 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 285 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 171 B/s wr, 1 op/s 2021-11-20T09:00:05.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:04 smithi093 conmon[33522]: audit 2021-11-20T09:00:03.972308+0000 mon.smithi093 (mon.0) 762 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T09:00:05.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:04 smithi093 conmon[33522]: audit 2021-11-20T09:00:04.513202+0000 mon.smithi093 (mon.0) 763 : audit [DBG] from='client.? 172.21.15.93:0/2023429582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:05.526 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:05.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- 172.21.15.93:0/952947478 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 msgr2=0x7fe7480f4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:05.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 --2- 172.21.15.93:0/952947478 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f4cf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fe738004660 tx=0x7fe738009b30).stop 2021-11-20T09:00:05.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- 172.21.15.93:0/952947478 shutdown_connections 2021-11-20T09:00:05.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 --2- 172.21.15.93:0/952947478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7480f5270 0x7fe7480f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:05.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 --2- 172.21.15.93:0/952947478 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f4cf0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:05.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- 172.21.15.93:0/952947478 >> 172.21.15.93:0/952947478 conn(0x7fe7480f0330 msgr2=0x7fe7480f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:05.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- 172.21.15.93:0/952947478 shutdown_connections 2021-11-20T09:00:05.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- 172.21.15.93:0/952947478 wait complete. 2021-11-20T09:00:05.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 Processor -- start 2021-11-20T09:00:05.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- start start 2021-11-20T09:00:05.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:05.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7480f5270 0x7fe7480f5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:05.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480f5ea0 con 0x7fe7480f48d0 2021-11-20T09:00:05.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.651+0000 7fe747fff700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480f5fe0 con 0x7fe7480f5270 2021-11-20T09:00:05.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe746ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:05.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe746ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44114/0 (socket says 172.21.15.93:44114) 2021-11-20T09:00:05.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe746ffd700 1 -- 172.21.15.93:0/3271870116 learned_addr learned my addr 172.21.15.93:0/3271870116 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:05.659 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe73e7fc700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7480f5270 0x7fe7480f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:05.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe746ffd700 1 -- 172.21.15.93:0/3271870116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7480f5270 msgr2=0x7fe7480f5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:05.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe746ffd700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7480f5270 0x7fe7480f5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:05.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe746ffd700 1 -- 172.21.15.93:0/3271870116 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738005040 con 0x7fe7480f48d0 2021-11-20T09:00:05.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe746ffd700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f74c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fe738009f50 tx=0x7fe738009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:05.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe73ffff700 1 -- 172.21.15.93:0/3271870116 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe738018070 con 0x7fe7480f48d0 2021-11-20T09:00:05.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe73ffff700 1 -- 172.21.15.93:0/3271870116 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe738013410 con 0x7fe7480f48d0 2021-11-20T09:00:05.661 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7480f6260 con 0x7fe7480f48d0 2021-11-20T09:00:05.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe73ffff700 1 -- 172.21.15.93:0/3271870116 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe73801c3c0 con 0x7fe7480f48d0 2021-11-20T09:00:05.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.652+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7480f67c0 con 0x7fe7480f48d0 2021-11-20T09:00:05.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.653+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe748045ad0 con 0x7fe7480f48d0 2021-11-20T09:00:05.662 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.653+0000 7fe73ffff700 1 -- 172.21.15.93:0/3271870116 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe73801d420 con 0x7fe7480f48d0 2021-11-20T09:00:05.663 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.654+0000 7fe73ffff700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe73405c300 0x7fe73405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:05.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.654+0000 7fe73ffff700 1 -- 172.21.15.93:0/3271870116 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe738077ef0 con 0x7fe7480f48d0 2021-11-20T09:00:05.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.654+0000 7fe73e7fc700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe73405c300 0x7fe73405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:05.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.654+0000 7fe73e7fc700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe73405c300 0x7fe73405e7c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe7480f56f0 tx=0x7fe73000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:05.664 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.658+0000 7fe73ffff700 1 -- 172.21.15.93:0/3271870116 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe73804a800 con 0x7fe7480f48d0 2021-11-20T09:00:05.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.841+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7480fef50 con 0x7fe7480f48d0 2021-11-20T09:00:05.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.841+0000 7fe73ffff700 1 -- 172.21.15.93:0/3271870116 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7fe73804de20 con 0x7fe7480f48d0 2021-11-20T09:00:05.844 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:05.845 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:05.846 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe73405c300 msgr2=0x7fe73405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:05.847 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe73405c300 0x7fe73405e7c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe7480f56f0 tx=0x7fe73000b040).stop 2021-11-20T09:00:05.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 msgr2=0x7fe7480f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:05.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f74c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fe738009f50 tx=0x7fe738009b30).stop 2021-11-20T09:00:05.848 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 shutdown_connections 2021-11-20T09:00:05.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe73405c300 0x7fe73405e7c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:05.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480f48d0 0x7fe7480f74c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:05.849 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 --2- 172.21.15.93:0/3271870116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7480f5270 0x7fe7480f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:05.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 >> 172.21.15.93:0/3271870116 conn(0x7fe7480f0330 msgr2=0x7fe7480fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:05.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 shutdown_connections 2021-11-20T09:00:05.850 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:05.843+0000 7fe747fff700 1 -- 172.21.15.93:0/3271870116 wait complete. 2021-11-20T09:00:05.851 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:05.854 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:05.855 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:06.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:05 smithi093 conmon[33522]: audit 2021-11-20T09:00:05.842107+0000 mon.smithi093 2021-11-20T09:00:06.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:05 smithi093 conmon[33522]: (mon.0) 764 : audit [DBG] from='client.? 172.21.15.93:0/3271870116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:06.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:05 smithi102 conmon[36472]: audit 2021-11-20T09:00:05.842107+0000 mon.smithi093 (mon.0) 764 2021-11-20T09:00:06.432 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:05 smithi102 conmon[36472]: : audit [DBG] from='client.? 172.21.15.93:0/3271870116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:06.859 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:06.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 -- 172.21.15.93:0/3281272965 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f65d0 msgr2=0x7f33100f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:06.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 --2- 172.21.15.93:0/3281272965 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f65d0 0x7f33100f69f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f330c004660 tx=0x7f330c009b30).stop 2021-11-20T09:00:06.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 -- 172.21.15.93:0/3281272965 shutdown_connections 2021-11-20T09:00:06.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 --2- 172.21.15.93:0/3281272965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f7870 0x7f33100f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:06.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 --2- 172.21.15.93:0/3281272965 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f65d0 0x7f33100f69f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:06.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 -- 172.21.15.93:0/3281272965 >> 172.21.15.93:0/3281272965 conn(0x7f33100f1d10 msgr2=0x7f33100f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:06.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 -- 172.21.15.93:0/3281272965 shutdown_connections 2021-11-20T09:00:06.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.986+0000 7f331681e700 1 -- 172.21.15.93:0/3281272965 wait complete. 2021-11-20T09:00:06.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331681e700 1 Processor -- start 2021-11-20T09:00:06.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331681e700 1 -- start start 2021-11-20T09:00:06.997 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331681e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f65d0 0x7f3310106d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:06.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331681e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f7870 0x7f3310107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:06.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331681e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3310107850 con 0x7f33100f7870 2021-11-20T09:00:06.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331681e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3310108060 con 0x7f33100f65d0 2021-11-20T09:00:06.998 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331501b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f7870 0x7f3310107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:06.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331501b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f7870 0x7f3310107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44124/0 (socket says 172.21.15.93:44124) 2021-11-20T09:00:06.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331501b700 1 -- 172.21.15.93:0/800203452 learned_addr learned my addr 172.21.15.93:0/800203452 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:06.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331581c700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f65d0 0x7f3310106d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:06.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331501b700 1 -- 172.21.15.93:0/800203452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f65d0 msgr2=0x7f3310106d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:07.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331501b700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f65d0 0x7f3310106d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:07.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.987+0000 7f331501b700 1 -- 172.21.15.93:0/800203452 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c005040 con 0x7f33100f7870 2021-11-20T09:00:07.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.988+0000 7f331501b700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f7870 0x7f3310107250 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f3300002700 tx=0x7f3300009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:07.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.988+0000 7f33067fc700 1 -- 172.21.15.93:0/800203452 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3300009320 con 0x7f33100f7870 2021-11-20T09:00:07.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.988+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33101082e0 con 0x7f33100f7870 2021-11-20T09:00:07.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.988+0000 7f33067fc700 1 -- 172.21.15.93:0/800203452 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f330000f040 con 0x7f33100f7870 2021-11-20T09:00:07.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.988+0000 7f33067fc700 1 -- 172.21.15.93:0/800203452 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3300014870 con 0x7f33100f7870 2021-11-20T09:00:07.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.988+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33101088a0 con 0x7f33100f7870 2021-11-20T09:00:07.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.989+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33100f59c0 con 0x7f33100f7870 2021-11-20T09:00:07.002 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.989+0000 7f33067fc700 1 -- 172.21.15.93:0/800203452 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3300009480 con 0x7f33100f7870 2021-11-20T09:00:07.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.989+0000 7f33067fc700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c300 0x7f32fc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:07.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.989+0000 7f33067fc700 1 -- 172.21.15.93:0/800203452 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3300076500 con 0x7f33100f7870 2021-11-20T09:00:07.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.990+0000 7f331581c700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c300 0x7f32fc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:07.003 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.990+0000 7f331581c700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c300 0x7f32fc05e7c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3310106ad0 tx=0x7f330c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:07.004 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:06.993+0000 7f33067fc700 1 -- 172.21.15.93:0/800203452 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3300048e10 con 0x7f33100f7870 2021-11-20T09:00:07.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.177+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3310045ad0 con 0x7f33100f7870 2021-11-20T09:00:07.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.178+0000 7f33067fc700 1 -- 172.21.15.93:0/800203452 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f330004c430 con 0x7f33100f7870 2021-11-20T09:00:07.177 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:07.178 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:07.179 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c300 msgr2=0x7f32fc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:07.179 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c300 0x7f32fc05e7c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3310106ad0 tx=0x7f330c009b30).stop 2021-11-20T09:00:07.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f7870 msgr2=0x7f3310107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:07.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f7870 0x7f3310107250 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f3300002700 tx=0x7f3300009e00).stop 2021-11-20T09:00:07.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 shutdown_connections 2021-11-20T09:00:07.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f32fc05c300 0x7f32fc05e7c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:07.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33100f65d0 0x7f3310106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:07.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 --2- 172.21.15.93:0/800203452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33100f7870 0x7f3310107250 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:07.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 >> 172.21.15.93:0/800203452 conn(0x7f33100f1d10 msgr2=0x7f33100fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:07.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 shutdown_connections 2021-11-20T09:00:07.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:07.179+0000 7f331681e700 1 -- 172.21.15.93:0/800203452 wait complete. 2021-11-20T09:00:07.182 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:07.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:06 smithi093 conmon[33522]: cluster 2021-11-20T09:00:05.570344+0000 mgr.smithi093.qdikkm (mgr.14162) 286 : cluster 2021-11-20T09:00:07.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:06 smithi093 conmon[33522]: [DBG] pgmap v260: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:07.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:07.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:07.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:06 smithi102 conmon[36472]: cluster 2021-11-20T09:00:05.570344+0000 mgr.smithi093.qdikkm (mgr.14162) 286 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:07.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:06 smithi102 conmon[36472]: 2021-11-20T09:00:08.193 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:08.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- 172.21.15.93:0/3745980650 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f6820 msgr2=0x7f740c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 --2- 172.21.15.93:0/3745980650 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f6820 0x7f740c0f6c80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7400004660 tx=0x7f7400009b30).stop 2021-11-20T09:00:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- 172.21.15.93:0/3745980650 shutdown_connections 2021-11-20T09:00:08.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 --2- 172.21.15.93:0/3745980650 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f6820 0x7f740c0f6c80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 --2- 172.21.15.93:0/3745980650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f740c0f5580 0x7f740c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- 172.21.15.93:0/3745980650 >> 172.21.15.93:0/3745980650 conn(0x7f740c0f0ce0 msgr2=0x7f740c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- 172.21.15.93:0/3745980650 shutdown_connections 2021-11-20T09:00:08.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- 172.21.15.93:0/3745980650 wait complete. 2021-11-20T09:00:08.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 Processor -- start 2021-11-20T09:00:08.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- start start 2021-11-20T09:00:08.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f5580 0x7f740c105d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:08.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f740c0f6820 0x7f740c106280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:08.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c106830 con 0x7f740c0f5580 2021-11-20T09:00:08.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f7410ee4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c106970 con 0x7f740c0f6820 2021-11-20T09:00:08.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f740a59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f5580 0x7f740c105d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:08.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f740a59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f5580 0x7f740c105d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44132/0 (socket says 172.21.15.93:44132) 2021-11-20T09:00:08.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.320+0000 7f740a59c700 1 -- 172.21.15.93:0/2055547105 learned_addr learned my addr 172.21.15.93:0/2055547105 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:08.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f7409d9b700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f740c0f6820 0x7f740c106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:08.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f740a59c700 1 -- 172.21.15.93:0/2055547105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f740c0f6820 msgr2=0x7f740c106280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:08.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f740a59c700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f740c0f6820 0x7f740c106280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:08.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f740a59c700 1 -- 172.21.15.93:0/2055547105 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7400005040 con 0x7f740c0f5580 2021-11-20T09:00:08.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f7409d9b700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f740c0f6820 0x7f740c106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:00:08.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f740a59c700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f5580 0x7f740c105d90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f73fc002700 tx=0x7f73fc009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:08.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f73faffd700 1 -- 172.21.15.93:0/2055547105 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73fc009320 con 0x7f740c0f5580 2021-11-20T09:00:08.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f73faffd700 1 -- 172.21.15.93:0/2055547105 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73fc00f040 con 0x7f740c0f5580 2021-11-20T09:00:08.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740c10b410 con 0x7f740c0f5580 2021-11-20T09:00:08.335 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f73faffd700 1 -- 172.21.15.93:0/2055547105 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73fc014900 con 0x7f740c0f5580 2021-11-20T09:00:08.336 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.321+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740c10b9d0 con 0x7f740c0f5580 2021-11-20T09:00:08.336 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.322+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740c027a00 con 0x7f740c0f5580 2021-11-20T09:00:08.337 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.322+0000 7f73faffd700 1 -- 172.21.15.93:0/2055547105 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f73fc009480 con 0x7f740c0f5580 2021-11-20T09:00:08.337 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.323+0000 7f73faffd700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f73f405c2a0 0x7f73f405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:08.340 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.323+0000 7f73faffd700 1 -- 172.21.15.93:0/2055547105 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f73fc076650 con 0x7f740c0f5580 2021-11-20T09:00:08.341 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.323+0000 7f7409d9b700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f73f405c2a0 0x7f73f405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:08.342 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.323+0000 7f7409d9b700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f73f405c2a0 0x7f73f405e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7400005010 tx=0x7f7400009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:08.343 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.327+0000 7f73faffd700 1 -- 172.21.15.93:0/2055547105 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73fc048ee0 con 0x7f740c0f5580 2021-11-20T09:00:08.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:07 smithi102 conmon[36472]: audit 2021-11-20T09:00:07.178604+0000 mon.smithi093 (mon.0) 765 : audit [DBG] from='client.? 172.21.15.93:0/800203452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:08.460 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:07 smithi093 conmon[33522]: audit 2021-11-20T09:00:07.178604+0000 mon.smithi093 (mon.0) 765 : audit [DBG] from='client.? 172.21.15.93:0/800203452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:08.509 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.511+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f740c0fdd10 con 0x7f740c0f5580 2021-11-20T09:00:08.510 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.511+0000 7f73faffd700 1 -- 172.21.15.93:0/2055547105 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f73fc04c500 con 0x7f740c0f5580 2021-11-20T09:00:08.511 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:08.511 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:08.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f73f405c2a0 msgr2=0x7f73f405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:08.512 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f73f405c2a0 0x7f73f405e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7400005010 tx=0x7f7400009b30).stop 2021-11-20T09:00:08.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f5580 msgr2=0x7f740c105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:08.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f5580 0x7f740c105d90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f73fc002700 tx=0x7f73fc009d80).stop 2021-11-20T09:00:08.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 shutdown_connections 2021-11-20T09:00:08.513 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f73f405c2a0 0x7f73f405e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:08.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f740c0f5580 0x7f740c105d90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:08.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 --2- 172.21.15.93:0/2055547105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f740c0f6820 0x7f740c106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:08.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 >> 172.21.15.93:0/2055547105 conn(0x7f740c0f0ce0 msgr2=0x7f740c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:08.514 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 shutdown_connections 2021-11-20T09:00:08.515 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:08.513+0000 7f7410ee4700 1 -- 172.21.15.93:0/2055547105 wait complete. 2021-11-20T09:00:08.515 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:08.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:08.525 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:09.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:08 smithi102 conmon[36472]: cluster 2021-11-20T09:00:07.570752+0000 mgr.smithi093.qdikkm (mgr.14162) 287 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:09.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:08 smithi102 conmon[36472]: audit 2021-11-20T09:00:09.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:08 smithi102 conmon[36472]: 2021-11-20T09:00:08.512226+0000 mon.smithi093 (mon.0) 766 : audit [DBG] from='client.? 172.21.15.93:0/2055547105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:09.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:08 smithi093 conmon[33522]: cluster 2021-11-20T09:00:07.570752+0000 mgr.smithi093.qdikkm (mgr.14162) 287 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:09.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:08 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T09:00:09.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:08 smithi093 conmon[33522]: 09:00:08.512226+0000 mon.smithi093 (mon.0) 766 : audit [DBG] from='client.? 172.21.15.93:0/2055547105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:09.526 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:09.666 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 -- 172.21.15.93:0/3467627351 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 msgr2=0x7f7c700f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/3467627351 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c700f59c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f7c600099e0 tx=0x7f7c60004d10).stop 2021-11-20T09:00:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 -- 172.21.15.93:0/3467627351 shutdown_connections 2021-11-20T09:00:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/3467627351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c700f6840 0x7f7c700f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:09.667 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/3467627351 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c700f59c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 -- 172.21.15.93:0/3467627351 >> 172.21.15.93:0/3467627351 conn(0x7f7c700f0ce0 msgr2=0x7f7c700f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 -- 172.21.15.93:0/3467627351 shutdown_connections 2021-11-20T09:00:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.663+0000 7f7c75e6e700 1 -- 172.21.15.93:0/3467627351 wait complete. 2021-11-20T09:00:09.668 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c75e6e700 1 Processor -- start 2021-11-20T09:00:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c75e6e700 1 -- start start 2021-11-20T09:00:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c75e6e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c7005ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c75e6e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c700f6840 0x7f7c7010e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:09.669 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c75e6e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70107b40 con 0x7f7c700f55a0 2021-11-20T09:00:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c75e6e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7010e650 con 0x7f7c700f6840 2021-11-20T09:00:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c6f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c7005ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c6f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c7005ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44142/0 (socket says 172.21.15.93:44142) 2021-11-20T09:00:09.670 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c6f7fe700 1 -- 172.21.15.93:0/77480142 learned_addr learned my addr 172.21.15.93:0/77480142 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c6effd700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c700f6840 0x7f7c7010e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c6f7fe700 1 -- 172.21.15.93:0/77480142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c700f6840 msgr2=0x7f7c7010e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.664+0000 7f7c6f7fe700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c700f6840 0x7f7c7010e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:09.671 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.665+0000 7f7c6f7fe700 1 -- 172.21.15.93:0/77480142 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c60009640 con 0x7f7c700f55a0 2021-11-20T09:00:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.665+0000 7f7c6f7fe700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c7005ff60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f7c600078b0 tx=0x7f7c60004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.665+0000 7f7c67fff700 1 -- 172.21.15.93:0/77480142 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c6000e030 con 0x7f7c700f55a0 2021-11-20T09:00:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.665+0000 7f7c67fff700 1 -- 172.21.15.93:0/77480142 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c60007d50 con 0x7f7c700f55a0 2021-11-20T09:00:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.665+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7010e7f0 con 0x7f7c700f55a0 2021-11-20T09:00:09.672 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.665+0000 7f7c67fff700 1 -- 172.21.15.93:0/77480142 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c60018070 con 0x7f7c700f55a0 2021-11-20T09:00:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.665+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7010eca0 con 0x7f7c700f55a0 2021-11-20T09:00:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.666+0000 7f7c67fff700 1 -- 172.21.15.93:0/77480142 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7c600137e0 con 0x7f7c700f55a0 2021-11-20T09:00:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.666+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70046330 con 0x7f7c700f55a0 2021-11-20T09:00:09.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.666+0000 7f7c67fff700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7c5c05c2a0 0x7f7c5c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:09.674 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.667+0000 7f7c67fff700 1 -- 172.21.15.93:0/77480142 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7c60077ff0 con 0x7f7c700f55a0 2021-11-20T09:00:09.675 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.667+0000 7f7c6effd700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7c5c05c2a0 0x7f7c5c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:09.675 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.667+0000 7f7c6effd700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7c5c05c2a0 0x7f7c5c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7c58004c40 tx=0x7f7c5800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:09.676 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.671+0000 7f7c67fff700 1 -- 172.21.15.93:0/77480142 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c6004a9b0 con 0x7f7c700f55a0 2021-11-20T09:00:09.857 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.857+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c70027a00 con 0x7f7c700f55a0 2021-11-20T09:00:09.858 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.857+0000 7f7c67fff700 1 -- 172.21.15.93:0/77480142 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7f7c6004dfd0 con 0x7f7c700f55a0 2021-11-20T09:00:09.859 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:09.859 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:09.866 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7c5c05c2a0 msgr2=0x7f7c5c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:09.867 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7c5c05c2a0 0x7f7c5c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7c58004c40 tx=0x7f7c5800b040).stop 2021-11-20T09:00:09.867 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 msgr2=0x7f7c7005ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:09.868 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c7005ff60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f7c600078b0 tx=0x7f7c60004d10).stop 2021-11-20T09:00:09.868 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 shutdown_connections 2021-11-20T09:00:09.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7c5c05c2a0 0x7f7c5c05e760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:09.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c700f55a0 0x7f7c7005ff60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:09.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 --2- 172.21.15.93:0/77480142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c700f6840 0x7f7c7010e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:09.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 >> 172.21.15.93:0/77480142 conn(0x7f7c700f0ce0 msgr2=0x7f7c700fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:09.871 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 shutdown_connections 2021-11-20T09:00:09.871 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:09.859+0000 7f7c75e6e700 1 -- 172.21.15.93:0/77480142 wait complete. 2021-11-20T09:00:09.872 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:09.875 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:09.876 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:10.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:09 smithi093 conmon[33522]: audit 2021-11-20T09:00:09.858313+0000 mon.smithi093 (mon 2021-11-20T09:00:10.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:09 smithi093 conmon[33522]: .0) 767 : audit [DBG] from='client.? 172.21.15.93:0/77480142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:10.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:09 smithi102 conmon[36472]: audit 2021-11-20T09:00:09.858313+0000 mon.smithi093 (mon.0) 767 : audit [DBG] from='client.? 172.21.15.93:0/77480142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:10.877 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:11.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 -- 172.21.15.93:0/319009809 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 msgr2=0x7ff4400f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:11.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 --2- 172.21.15.93:0/319009809 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff4400f6fa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7ff430004660 tx=0x7ff430009b30).stop 2021-11-20T09:00:11.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 -- 172.21.15.93:0/319009809 shutdown_connections 2021-11-20T09:00:11.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 --2- 172.21.15.93:0/319009809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4400f7520 0x7ff4400f99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:11.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 --2- 172.21.15.93:0/319009809 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff4400f6fa0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:11.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 -- 172.21.15.93:0/319009809 >> 172.21.15.93:0/319009809 conn(0x7ff4400f0c10 msgr2=0x7ff4400f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:11.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 -- 172.21.15.93:0/319009809 shutdown_connections 2021-11-20T09:00:11.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 -- 172.21.15.93:0/319009809 wait complete. 2021-11-20T09:00:11.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 Processor -- start 2021-11-20T09:00:11.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.012+0000 7ff4450ae700 1 -- start start 2021-11-20T09:00:11.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff4450ae700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff44010d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:11.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff4450ae700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4400f7520 0x7ff44010d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:11.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff4450ae700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44010dd90 con 0x7ff4400604d0 2021-11-20T09:00:11.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff4450ae700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44010ded0 con 0x7ff4400f7520 2021-11-20T09:00:11.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff44010d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:11.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff44010d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44154/0 (socket says 172.21.15.93:44154) 2021-11-20T09:00:11.019 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43ffff700 1 -- 172.21.15.93:0/2054643182 learned_addr learned my addr 172.21.15.93:0/2054643182 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:11.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43f7fe700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4400f7520 0x7ff44010d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:11.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43ffff700 1 -- 172.21.15.93:0/2054643182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4400f7520 msgr2=0x7ff44010d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:11.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43ffff700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4400f7520 0x7ff44010d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:11.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43ffff700 1 -- 172.21.15.93:0/2054643182 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430005040 con 0x7ff4400604d0 2021-11-20T09:00:11.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43ffff700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff44010d2f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7ff430009f50 tx=0x7ff430009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:11.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43cff9700 1 -- 172.21.15.93:0/2054643182 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff430018070 con 0x7ff4400604d0 2021-11-20T09:00:11.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43cff9700 1 -- 172.21.15.93:0/2054643182 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff430013410 con 0x7ff4400604d0 2021-11-20T09:00:11.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44010e810 con 0x7ff4400604d0 2021-11-20T09:00:11.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff43cff9700 1 -- 172.21.15.93:0/2054643182 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff43001c3c0 con 0x7ff4400604d0 2021-11-20T09:00:11.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.013+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44010ed70 con 0x7ff4400604d0 2021-11-20T09:00:11.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.014+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff440045ad0 con 0x7ff4400604d0 2021-11-20T09:00:11.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.015+0000 7ff43cff9700 1 -- 172.21.15.93:0/2054643182 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff43001d420 con 0x7ff4400604d0 2021-11-20T09:00:11.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.015+0000 7ff43cff9700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff42805c300 0x7ff42805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:11.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.015+0000 7ff43cff9700 1 -- 172.21.15.93:0/2054643182 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff430077ef0 con 0x7ff4400604d0 2021-11-20T09:00:11.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.015+0000 7ff43f7fe700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff42805c300 0x7ff42805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:11.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.015+0000 7ff43f7fe700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff42805c300 0x7ff42805e7c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff434004c40 tx=0x7ff43400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:11.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.019+0000 7ff43cff9700 1 -- 172.21.15.93:0/2054643182 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff43004a800 con 0x7ff4400604d0 2021-11-20T09:00:11.202 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.202+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff440027a00 con 0x7ff4400604d0 2021-11-20T09:00:11.203 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.202+0000 7ff43cff9700 1 -- 172.21.15.93:0/2054643182 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2566 (secure 0 0 0) 0x7ff440027a00 con 0x7ff4400604d0 2021-11-20T09:00:11.207 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:11.207 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11},{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T08:59:54.598882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14638":{"gid":14638,"name":"cephfs.smithi093.ljzugc","rank":0,"incarnation":66,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.93:6827/3869812955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":3869812955},{"type":"v1","addr":"172.21.15.93:6827","nonce":3869812955}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:11.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff42805c300 msgr2=0x7ff42805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:11.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff42805c300 0x7ff42805e7c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff434004c40 tx=0x7ff43400b040).stop 2021-11-20T09:00:11.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 msgr2=0x7ff44010d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:11.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff44010d2f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7ff430009f50 tx=0x7ff430009b30).stop 2021-11-20T09:00:11.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 shutdown_connections 2021-11-20T09:00:11.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff42805c300 0x7ff42805e7c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:11.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4400604d0 0x7ff44010d2f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:11.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 --2- 172.21.15.93:0/2054643182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4400f7520 0x7ff44010d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:11.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 >> 172.21.15.93:0/2054643182 conn(0x7ff4400f0c10 msgr2=0x7ff4400f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:11.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 shutdown_connections 2021-11-20T09:00:11.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:11.204+0000 7ff4450ae700 1 -- 172.21.15.93:0/2054643182 wait complete. 2021-11-20T09:00:11.212 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 69 2021-11-20T09:00:11.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:10 smithi093 conmon[33522]: cluster 2021-11-20T09:00:09.571306+0000 mgr.smithi093.qdikkm (mgr.14162) 288 : cluster [DBG] 2021-11-20T09:00:11.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:10 smithi093 conmon[33522]: pgmap v262: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:00:11.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T08:59:54.598882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14638': {'gid': 14638, 'name': 'cephfs.smithi093.ljzugc', 'rank': 0, 'incarnation': 66, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.93:6827/3869812955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6826', 'nonce': 3869812955}, {'type': 'v1', 'addr': '172.21.15.93:6827', 'nonce': 3869812955}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:11.215 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14638:up:clientreplay 2021-11-20T09:00:11.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:10 smithi102 conmon[36472]: cluster 2021-11-20T09:00:09.571306+0000 mgr.smithi093.qdikkm (mgr.14162) 288 : cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:00:12.217 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:12.348 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 -- 172.21.15.93:0/3987320442 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f5580 msgr2=0x7f642c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:12.349 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 --2- 172.21.15.93:0/3987320442 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f5580 0x7f642c0f59a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6428004660 tx=0x7f6428009b30).stop 2021-11-20T09:00:12.349 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 -- 172.21.15.93:0/3987320442 shutdown_connections 2021-11-20T09:00:12.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 --2- 172.21.15.93:0/3987320442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c0f6820 0x7f642c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:12.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 --2- 172.21.15.93:0/3987320442 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f5580 0x7f642c0f59a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:12.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 -- 172.21.15.93:0/3987320442 >> 172.21.15.93:0/3987320442 conn(0x7f642c0f0ce0 msgr2=0x7f642c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:12.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 -- 172.21.15.93:0/3987320442 shutdown_connections 2021-11-20T09:00:12.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.345+0000 7f6434466700 1 -- 172.21.15.93:0/3987320442 wait complete. 2021-11-20T09:00:12.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 Processor -- start 2021-11-20T09:00:12.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 -- start start 2021-11-20T09:00:12.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c0f5580 0x7f642c105d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:12.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f6820 0x7f642c106270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:12.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c106870 con 0x7f642c0f6820 2021-11-20T09:00:12.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1069b0 con 0x7f642c0f5580 2021-11-20T09:00:12.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6431a01700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f6820 0x7f642c106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:12.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6431a01700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f6820 0x7f642c106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44166/0 (socket says 172.21.15.93:44166) 2021-11-20T09:00:12.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6431a01700 1 -- 172.21.15.93:0/2587692966 learned_addr learned my addr 172.21.15.93:0/2587692966 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:12.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6432202700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c0f5580 0x7f642c105d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:12.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6431a01700 1 -- 172.21.15.93:0/2587692966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c0f5580 msgr2=0x7f642c105d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:12.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6431a01700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c0f5580 0x7f642c105d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:12.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6431a01700 1 -- 172.21.15.93:0/2587692966 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6428005040 con 0x7f642c0f6820 2021-11-20T09:00:12.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6432202700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c0f5580 0x7f642c105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:00:12.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6431a01700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f6820 0x7f642c106270 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f641c002700 tx=0x7f641c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:12.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6422ffd700 1 -- 172.21.15.93:0/2587692966 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641c009320 con 0x7f642c0f6820 2021-11-20T09:00:12.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6422ffd700 1 -- 172.21.15.93:0/2587692966 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f641c00f040 con 0x7f642c0f6820 2021-11-20T09:00:12.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c10b3c0 con 0x7f642c0f6820 2021-11-20T09:00:12.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6422ffd700 1 -- 172.21.15.93:0/2587692966 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641c014820 con 0x7f642c0f6820 2021-11-20T09:00:12.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.346+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c10b980 con 0x7f642c0f6820 2021-11-20T09:00:12.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.347+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c027a00 con 0x7f642c0f6820 2021-11-20T09:00:12.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.348+0000 7f6422ffd700 1 -- 172.21.15.93:0/2587692966 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f641c0077b0 con 0x7f642c0f6820 2021-11-20T09:00:12.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.348+0000 7f6422ffd700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f641805c2a0 0x7f641805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:12.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.348+0000 7f6422ffd700 1 -- 172.21.15.93:0/2587692966 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f641c076640 con 0x7f642c0f6820 2021-11-20T09:00:12.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.348+0000 7f6432202700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f641805c2a0 0x7f641805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:12.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.349+0000 7f6432202700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f641805c2a0 0x7f641805e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6428009d60 tx=0x7f6428009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:12.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.352+0000 7f6422ffd700 1 -- 172.21.15.93:0/2587692966 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f641c048f00 con 0x7f642c0f6820 2021-11-20T09:00:12.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: audit 2021-11-20T09:00:11.203330+0000 mon.smithi093 (mon.0) 768 : audit 2021-11-20T09:00:12.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: [DBG] from='client.? 172.21.15.93:0/2054643182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:12.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: cluster 2021-11-20T09:00:11.595568+0000 mon.smithi093 (mon.0) 769 : cluster [WRN] Replacing daemon mds.cephfs.smithi093.ljzugc as rank 0 with standby daemon mds.cephfs.smithi102.hjodmi 2021-11-20T09:00:12.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: cluster 2021-11-20T09:00 2021-11-20T09:00:12.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: :11.595625+0000 mon.smithi093 (mon.0) 770 : cluster [INF] MDS daemon mds.cephfs.smithi093.ljzugc is removed because it is dead or otherwise unavailable. 2021-11-20T09:00:12.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: cluster 2021- 2021-11-20T09:00:12.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: 11-20T09:00:11.599628+0000 mon.smithi093 (mon.0) 771 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T09:00:12.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: cluster 2021 2021-11-20T09:00:12.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: -11-20T09:00:11.602540+0000 mon.smithi093 (mon. 2021-11-20T09:00:12.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:12 smithi102 conmon[36472]: 0) 772 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:replay} 1 up:standby 2021-11-20T09:00:12.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: audit 2021-11-20T09:00:11.203330+0000 mon.smithi093 (mon.0) 768 : audit [DBG] from='client.? 172.21.15.93:0/2054643182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:12.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: cluster 2021-11-20T09:00 2021-11-20T09:00:12.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: :11.595568+0000 mon.smithi093 (mon.0) 769 : cluster [WRN] Replacing daemon mds.cephfs.smithi093.ljzugc as rank 0 with standby daemon mds.cephfs.smithi102.hjodmi 2021-11-20T09:00:12.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: cluster 2021-11-20T09:00:11.595625+0000 mon.smithi093 (mon.0) 770 : cluster 2021-11-20T09:00:12.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: [INF] MDS daemon mds.cephfs.smithi093.ljzugc is removed because it is dead or otherwise unavailable. 2021-11-20T09:00:12.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: cluster 2021-11-20T09:00:12.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: 2021-11-20T09:00:11.599628+0000 mon.smithi093 2021-11-20T09:00:12.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: (mon.0 2021-11-20T09:00:12.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: ) 771 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T09:00:12.464 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: cluster 2021- 2021-11-20T09:00:12.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: 11-20T09:00:11.602540+0000 2021-11-20T09:00:12.465 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:11 smithi093 conmon[33522]: mon.smithi093 (mon.0) 772 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:replay} 1 up:standby 2021-11-20T09:00:12.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.537+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f642c0fdd10 con 0x7f642c0f6820 2021-11-20T09:00:12.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.537+0000 7f6422ffd700 1 -- 172.21.15.93:0/2587692966 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2185 (secure 0 0 0) 0x7f641c04c520 con 0x7f642c0f6820 2021-11-20T09:00:12.536 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:12.537 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:11.596011+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:replay","state_seq":2,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:12.538 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f641805c2a0 msgr2=0x7f641805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:12.538 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f641805c2a0 0x7f641805e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6428009d60 tx=0x7f6428009b30).stop 2021-11-20T09:00:12.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f6820 msgr2=0x7f642c106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:12.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f6820 0x7f642c106270 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f641c002700 tx=0x7f641c009e00).stop 2021-11-20T09:00:12.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 shutdown_connections 2021-11-20T09:00:12.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f641805c2a0 0x7f641805e760 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:12.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c0f5580 0x7f642c105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:12.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 --2- 172.21.15.93:0/2587692966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642c0f6820 0x7f642c106270 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:12.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 >> 172.21.15.93:0/2587692966 conn(0x7f642c0f0ce0 msgr2=0x7f642c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:12.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 shutdown_connections 2021-11-20T09:00:12.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:12.539+0000 7f6434466700 1 -- 172.21.15.93:0/2587692966 wait complete. 2021-11-20T09:00:12.541 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 70 2021-11-20T09:00:12.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:11.596011+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:12.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:replay 2021-11-20T09:00:13.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:13 smithi102 conmon[36472]: cluster 2021-11-20T09:00:11.571974+0000 mgr.smithi093.qdikkm (mgr.14162) 289 : cluster [DBG] pgmap v263: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:00:13.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:13 smithi102 conmon[36472]: audit 2021-11-20T09:00:12.538095+0000 mon.smithi093 (mon.0) 773 : audit [DBG] from='client.? 172.21.15.93:0/2587692966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:13.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:13 smithi102 conmon[36472]: cluster 2021-11-20T09:00:12.604709+0000 mon.smithi093 ( 2021-11-20T09:00:13.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:13 smithi102 conmon[36472]: mon.0) 774 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:reconnect 2021-11-20T09:00:13.420 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:13 smithi102 conmon[36472]: cluster 2021 2021-11-20T09:00:13.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:13 smithi102 conmon[36472]: -11-20T09:00:12.604769+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:13.421 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:13 smithi102 conmon[36472]: 775 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:reconnect} 1 up:standby 2021-11-20T09:00:13.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:13 smithi093 conmon[33522]: cluster 2021-11-20T09:00:11.571974+0000 mgr.smithi093.qdikkm (mgr.14162) 289 : 2021-11-20T09:00:13.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:13 smithi093 conmon[33522]: cluster [DBG] pgmap v263: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:00:13.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:13 smithi093 conmon[33522]: audit 2021-11-20T09:00:12.538095+0000 mon.smithi093 (mon.0) 773 : audit [DBG] from='client.? 172.21.15.93:0/2587692966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:13.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:13 smithi093 conmon[33522]: cluster 2021- 2021-11-20T09:00:13.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:13 smithi093 conmon[33522]: 11-20T09:00:12.604709+0000 mon.smithi093 (mon.0) 774 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:reconnect 2021-11-20T09:00:13.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:13 smithi093 conmon[33522]: cluster 2021-11-20T09: 2021-11-20T09:00:13.463 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:13 smithi093 conmon[33522]: 00:12.604769+0000 mon.smithi093 (mon.0) 775 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:reconnect} 1 up:standby 2021-11-20T09:00:13.551 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:13.680 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.677+0000 7f0d74015700 1 -- 172.21.15.93:0/2494758970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 msgr2=0x7f0d6c0f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:13.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.677+0000 7f0d74015700 1 --2- 172.21.15.93:0/2494758970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c0f9250 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f0d68004660 tx=0x7f0d68009b30).stop 2021-11-20T09:00:13.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.677+0000 7f0d74015700 1 -- 172.21.15.93:0/2494758970 shutdown_connections 2021-11-20T09:00:13.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.677+0000 7f0d74015700 1 --2- 172.21.15.93:0/2494758970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d6c0f97d0 0x7f0d6c0fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:13.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.677+0000 7f0d74015700 1 --2- 172.21.15.93:0/2494758970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c0f9250 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:13.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.677+0000 7f0d74015700 1 -- 172.21.15.93:0/2494758970 >> 172.21.15.93:0/2494758970 conn(0x7f0d6c0f0cb0 msgr2=0x7f0d6c0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:13.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 -- 172.21.15.93:0/2494758970 shutdown_connections 2021-11-20T09:00:13.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 -- 172.21.15.93:0/2494758970 wait complete. 2021-11-20T09:00:13.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 Processor -- start 2021-11-20T09:00:13.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 -- start start 2021-11-20T09:00:13.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:13.683 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d6c0f97d0 0x7f0d6c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:13.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c107950 con 0x7f0d6c060710 2021-11-20T09:00:13.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d74015700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c107a90 con 0x7f0d6c0f97d0 2021-11-20T09:00:13.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d71db1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:13.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d71db1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44176/0 (socket says 172.21.15.93:44176) 2021-11-20T09:00:13.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d71db1700 1 -- 172.21.15.93:0/2129701196 learned_addr learned my addr 172.21.15.93:0/2129701196 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:13.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.678+0000 7f0d715b0700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d6c0f97d0 0x7f0d6c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:13.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d71db1700 1 -- 172.21.15.93:0/2129701196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d6c0f97d0 msgr2=0x7f0d6c1073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:13.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d71db1700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d6c0f97d0 0x7f0d6c1073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:13.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d71db1700 1 -- 172.21.15.93:0/2129701196 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d68005040 con 0x7f0d6c060710 2021-11-20T09:00:13.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d715b0700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d6c0f97d0 0x7f0d6c1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:00:13.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d71db1700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c106eb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f0d68009830 tx=0x7f0d68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:13.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d627fc700 1 -- 172.21.15.93:0/2129701196 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d68018070 con 0x7f0d6c060710 2021-11-20T09:00:13.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d627fc700 1 -- 172.21.15.93:0/2129701196 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d68013410 con 0x7f0d6c060710 2021-11-20T09:00:13.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c10c4d0 con 0x7f0d6c060710 2021-11-20T09:00:13.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d627fc700 1 -- 172.21.15.93:0/2129701196 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d6801c620 con 0x7f0d6c060710 2021-11-20T09:00:13.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.679+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c10ca90 con 0x7f0d6c060710 2021-11-20T09:00:13.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.680+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c027a00 con 0x7f0d6c060710 2021-11-20T09:00:13.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.680+0000 7f0d627fc700 1 -- 172.21.15.93:0/2129701196 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0d6801c890 con 0x7f0d6c060710 2021-11-20T09:00:13.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.680+0000 7f0d627fc700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d5805c2f0 0x7f0d5805e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:13.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.681+0000 7f0d627fc700 1 -- 172.21.15.93:0/2129701196 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f0d68078170 con 0x7f0d6c060710 2021-11-20T09:00:13.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.681+0000 7f0d715b0700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d5805c2f0 0x7f0d5805e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:13.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.681+0000 7f0d715b0700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d5805c2f0 0x7f0d5805e7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0d5c0041d0 tx=0x7f0d5c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:13.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.684+0000 7f0d627fc700 1 -- 172.21.15.93:0/2129701196 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d6804a990 con 0x7f0d6c060710 2021-11-20T09:00:13.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.869+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d6c0fee20 con 0x7f0d6c060710 2021-11-20T09:00:13.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.869+0000 7f0d627fc700 1 -- 172.21.15.93:0/2129701196 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2186 (secure 0 0 0) 0x7f0d6804dfb0 con 0x7f0d6c060710 2021-11-20T09:00:13.871 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:13.871 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:13.602881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:rejoin","state_seq":53,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:13.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d5805c2f0 msgr2=0x7f0d5805e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:13.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d5805c2f0 0x7f0d5805e7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0d5c0041d0 tx=0x7f0d5c009040).stop 2021-11-20T09:00:13.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 msgr2=0x7f0d6c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:13.875 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c106eb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f0d68009830 tx=0x7f0d68009b30).stop 2021-11-20T09:00:13.875 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 shutdown_connections 2021-11-20T09:00:13.876 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0d5805c2f0 0x7f0d5805e7b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:13.876 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c060710 0x7f0d6c106eb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:13.876 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 --2- 172.21.15.93:0/2129701196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d6c0f97d0 0x7f0d6c1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:13.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 >> 172.21.15.93:0/2129701196 conn(0x7f0d6c0f0cb0 msgr2=0x7f0d6c0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:13.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 shutdown_connections 2021-11-20T09:00:13.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:13.871+0000 7f0d74015700 1 -- 172.21.15.93:0/2129701196 wait complete. 2021-11-20T09:00:13.878 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 72 2021-11-20T09:00:13.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:13.602881+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:rejoin', 'state_seq': 53, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:13.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:rejoin 2021-11-20T09:00:14.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:14 smithi093 conmon[33522]: cluster 2021-11-20T09:00:12.624691+0000 mds.cephfs.smithi102.hjodmi (mds.0) 2021-11-20T09:00:14.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:14 smithi093 conmon[33522]: 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:00:14.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:14 smithi093 conmon[33522]: cluster 2021-11-20T09:00:13.606447+0000 mon.smithi093 (mon.0) 776 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:rejoin 2021-11-20T09:00:14.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:14 smithi093 conmon[33522]: cluster 2021-11-20T09:00:13.606512+0000 mon.smithi093 (mon.0) 777 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:rejoin} 1 up:standby 2021-11-20T09:00:14.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:14 smithi093 conmon[33522]: audit 2021-11-20T09:00:13.871238+0000 mon.smithi093 (mon.0) 778 : audit [DBG] from='client.? 172.21.15.93:0/2129701196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:14.412 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:14 smithi102 conmon[36472]: cluster 2021-11-20T09:00:12.624691+0000 mds.cephfs.smithi102.hjodmi (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:00:14.412 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:14 smithi102 conmon[36472]: cluster 2021-11-20T09:00 2021-11-20T09:00:14.412 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:14 smithi102 conmon[36472]: :13.606447+0000 mon.smithi093 (mon.0) 776 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:rejoin 2021-11-20T09:00:14.413 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:14 smithi102 conmon[36472]: cluster 2021-11-20T09:00:13.606512+0000 mon.smithi093 (mon.0) 777 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:rejoin} 1 up:standby 2021-11-20T09:00:14.413 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:14 smithi102 conmon[36472]: audit 2021-11-20T09:00:14.413 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:14 smithi102 conmon[36472]: 2021-11-20T09:00:13.871238+0000 mon.smithi093 (mon.0) 778 : audit [DBG] from='client.? 172.21.15.93:0/2129701196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:14.884 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:15.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 -- 172.21.15.93:0/57143782 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 msgr2=0x7fe01c0f4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:15.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 --2- 172.21.15.93:0/57143782 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c0f4e40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fe00c004660 tx=0x7fe00c009b30).stop 2021-11-20T09:00:15.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 -- 172.21.15.93:0/57143782 shutdown_connections 2021-11-20T09:00:15.014 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 --2- 172.21.15.93:0/57143782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe01c0f53c0 0x7fe01c0fbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:15.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 --2- 172.21.15.93:0/57143782 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c0f4e40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:15.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 -- 172.21.15.93:0/57143782 >> 172.21.15.93:0/57143782 conn(0x7fe01c0f0440 msgr2=0x7fe01c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:15.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 -- 172.21.15.93:0/57143782 shutdown_connections 2021-11-20T09:00:15.015 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 -- 172.21.15.93:0/57143782 wait complete. 2021-11-20T09:00:15.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 Processor -- start 2021-11-20T09:00:15.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 -- start start 2021-11-20T09:00:15.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:15.016 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.010+0000 7fe02249c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe01c0f53c0 0x7fe01c107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:15.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe02249c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c107940 con 0x7fe01c0f4a20 2021-11-20T09:00:15.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe02249c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c107a80 con 0x7fe01c0f53c0 2021-11-20T09:00:15.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:15.017 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44186/0 (socket says 172.21.15.93:44186) 2021-11-20T09:00:15.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01bfff700 1 -- 172.21.15.93:0/2311805041 learned_addr learned my addr 172.21.15.93:0/2311805041 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:15.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01b7fe700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe01c0f53c0 0x7fe01c107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:15.018 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01bfff700 1 -- 172.21.15.93:0/2311805041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe01c0f53c0 msgr2=0x7fe01c107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:15.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01bfff700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe01c0f53c0 0x7fe01c107390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:15.020 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01bfff700 1 -- 172.21.15.93:0/2311805041 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c005040 con 0x7fe01c0f4a20 2021-11-20T09:00:15.021 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe01bfff700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c106ea0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fe00c009f50 tx=0x7fe00c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:15.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe018ff9700 1 -- 172.21.15.93:0/2311805041 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe00c018070 con 0x7fe01c0f4a20 2021-11-20T09:00:15.022 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe018ff9700 1 -- 172.21.15.93:0/2311805041 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe00c013410 con 0x7fe01c0f4a20 2021-11-20T09:00:15.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe018ff9700 1 -- 172.21.15.93:0/2311805041 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe00c01c3c0 con 0x7fe01c0f4a20 2021-11-20T09:00:15.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c10c4c0 con 0x7fe01c0f4a20 2021-11-20T09:00:15.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.011+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c10ca80 con 0x7fe01c0f4a20 2021-11-20T09:00:15.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.012+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c027a00 con 0x7fe01c0f4a20 2021-11-20T09:00:15.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.012+0000 7fe018ff9700 1 -- 172.21.15.93:0/2311805041 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe00c01d420 con 0x7fe01c0f4a20 2021-11-20T09:00:15.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.013+0000 7fe018ff9700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe00405c2a0 0x7fe00405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:15.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.013+0000 7fe018ff9700 1 -- 172.21.15.93:0/2311805041 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe00c077fc0 con 0x7fe01c0f4a20 2021-11-20T09:00:15.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.013+0000 7fe01b7fe700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe00405c2a0 0x7fe00405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:15.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.013+0000 7fe01b7fe700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe00405c2a0 0x7fe00405e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe01c0f5840 tx=0x7fe010009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:15.030 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.017+0000 7fe018ff9700 1 -- 172.21.15.93:0/2311805041 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe00c04a880 con 0x7fe01c0f4a20 2021-11-20T09:00:15.174 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:15 smithi093 conmon[33522]: cluster 2021-11-20T09:00:13.572327+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T09:00:15.174 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:15 smithi093 conmon[33522]: ) 290 : cluster [DBG] pgmap v265: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T09:00:15.174 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:15 smithi093 conmon[33522]: cluster 2021-11-20T09:00:14.608053+0000 mon.smithi093 (mon.0) 779 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:clientreplay 2021-11-20T09:00:15.175 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:15 smithi093 conmon[33522]: cluster 2021-11-20T09:00:14.608124+0000 2021-11-20T09:00:15.175 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:15 smithi093 conmon[33522]: mon.smithi093 (mon.0) 780 : cluster 2021-11-20T09:00:15.175 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:15 smithi093 conmon[33522]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:00:15.207 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.207+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe01c0fee40 con 0x7fe01c0f4a20 2021-11-20T09:00:15.208 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.207+0000 7fe018ff9700 1 -- 172.21.15.93:0/2311805041 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7fe00c04dea0 con 0x7fe01c0f4a20 2021-11-20T09:00:15.208 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:15.208 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:15.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe00405c2a0 msgr2=0x7fe00405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:15.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe00405c2a0 0x7fe00405e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe01c0f5840 tx=0x7fe010009c00).stop 2021-11-20T09:00:15.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 msgr2=0x7fe01c106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:15.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c106ea0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fe00c009f50 tx=0x7fe00c009b30).stop 2021-11-20T09:00:15.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 shutdown_connections 2021-11-20T09:00:15.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe00405c2a0 0x7fe00405e760 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:15.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe01c0f4a20 0x7fe01c106ea0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:15.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 --2- 172.21.15.93:0/2311805041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe01c0f53c0 0x7fe01c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:15.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 >> 172.21.15.93:0/2311805041 conn(0x7fe01c0f0440 msgr2=0x7fe01c0f84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:15.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 shutdown_connections 2021-11-20T09:00:15.213 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:15.209+0000 7fe02249c700 1 -- 172.21.15.93:0/2311805041 wait complete. 2021-11-20T09:00:15.214 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:15.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:15.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:15.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:15 smithi102 conmon[36472]: cluster 2021-11-20T09:00:13.572327+0000 mgr.smithi093.qdikkm (mgr.14162) 290 : cluster [DBG] pgmap v265: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T09:00:15.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:15 smithi102 conmon[36472]: cluster 2021-11-20T 2021-11-20T09:00:15.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:15 smithi102 conmon[36472]: 09:00:14.608053+0000 mon.smithi093 (mon.0) 779 : cluster [DBG] mds.? [v2:172.21.15.102:6826/1123372428,v1:172.21.15.102:6827/1123372428] up:clientreplay 2021-11-20T09:00:15.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:15 smithi102 conmon[36472]: cluster 2021-11-20T09:00:14.608124+0000 mon.smithi093 (mon.0) 780 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:00:16.222 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:16.248 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:16 smithi093 conmon[33522]: audit 2021-11-20T09:00:15.209563+0000 mon.smithi093 (mon.0) 781 2021-11-20T09:00:16.249 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:16 smithi093 conmon[33522]: : audit [DBG] from='client.? 172.21.15.93:0/2311805041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:16.350 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 -- 172.21.15.93:0/235109850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f4d20 msgr2=0x7fabb00f5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:16.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 --2- 172.21.15.93:0/235109850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f4d20 0x7fabb00f5140 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7faba0004770 tx=0x7faba000ac40).stop 2021-11-20T09:00:16.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 -- 172.21.15.93:0/235109850 shutdown_connections 2021-11-20T09:00:16.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 --2- 172.21.15.93:0/235109850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabb00f5fc0 0x7fabb00f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:16.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 --2- 172.21.15.93:0/235109850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f4d20 0x7fabb00f5140 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:16.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 -- 172.21.15.93:0/235109850 >> 172.21.15.93:0/235109850 conn(0x7fabb00f0440 msgr2=0x7fabb00f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:16.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 -- 172.21.15.93:0/235109850 shutdown_connections 2021-11-20T09:00:16.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.345+0000 7fabb55c5700 1 -- 172.21.15.93:0/235109850 wait complete. 2021-11-20T09:00:16.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabb55c5700 1 Processor -- start 2021-11-20T09:00:16.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabb55c5700 1 -- start start 2021-11-20T09:00:16.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabb55c5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabb00f4d20 0x7fabb0106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:16.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabb55c5700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f5fc0 0x7fabb0107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:16.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabb55c5700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb0107900 con 0x7fabb00f5fc0 2021-11-20T09:00:16.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabb55c5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb0107a40 con 0x7fabb00f4d20 2021-11-20T09:00:16.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabae7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f5fc0 0x7fabb0107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:16.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabae7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f5fc0 0x7fabb0107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44204/0 (socket says 172.21.15.93:44204) 2021-11-20T09:00:16.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabae7fc700 1 -- 172.21.15.93:0/2587439944 learned_addr learned my addr 172.21.15.93:0/2587439944 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:16.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabaeffd700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabb00f4d20 0x7fabb0106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:16.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabae7fc700 1 -- 172.21.15.93:0/2587439944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabb00f4d20 msgr2=0x7fabb0106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:16.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabae7fc700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabb00f4d20 0x7fabb0106e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:16.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabae7fc700 1 -- 172.21.15.93:0/2587439944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba0005040 con 0x7fabb00f5fc0 2021-11-20T09:00:16.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fabae7fc700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f5fc0 0x7fabb0107350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7faba400e7f0 tx=0x7faba400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:16.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.346+0000 7fab977fe700 1 -- 172.21.15.93:0/2587439944 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faba40092e0 con 0x7fabb00f5fc0 2021-11-20T09:00:16.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.347+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb010c4e0 con 0x7fabb00f5fc0 2021-11-20T09:00:16.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.347+0000 7fab977fe700 1 -- 172.21.15.93:0/2587439944 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faba400f040 con 0x7fabb00f5fc0 2021-11-20T09:00:16.361 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.347+0000 7fab977fe700 1 -- 172.21.15.93:0/2587439944 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faba4014750 con 0x7fabb00f5fc0 2021-11-20T09:00:16.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.347+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb010caa0 con 0x7fabb00f5fc0 2021-11-20T09:00:16.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.348+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb0027a00 con 0x7fabb00f5fc0 2021-11-20T09:00:16.362 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.348+0000 7fab977fe700 1 -- 172.21.15.93:0/2587439944 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faba401b030 con 0x7fabb00f5fc0 2021-11-20T09:00:16.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.348+0000 7fab977fe700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fab9805c2a0 0x7fab9805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:16.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.348+0000 7fab977fe700 1 -- 172.21.15.93:0/2587439944 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7faba4076060 con 0x7fabb00f5fc0 2021-11-20T09:00:16.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.348+0000 7fabaeffd700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fab9805c2a0 0x7fab9805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:16.375 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.349+0000 7fabaeffd700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fab9805c2a0 0x7fab9805e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7faba00144e0 tx=0x7faba0016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:16.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.352+0000 7fab977fe700 1 -- 172.21.15.93:0/2587439944 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faba4048970 con 0x7fabb00f5fc0 2021-11-20T09:00:16.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:16 smithi102 conmon[36472]: audit 2021-11-20T09:00:15.209563+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:16.419 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:16 smithi102 conmon[36472]: 781 : audit [DBG] from='client.? 172.21.15.93:0/2311805041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:16.539 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.538+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabb00fcc20 con 0x7fabb00f5fc0 2021-11-20T09:00:16.540 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.539+0000 7fab977fe700 1 -- 172.21.15.93:0/2587439944 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7faba4019020 con 0x7fabb00f5fc0 2021-11-20T09:00:16.541 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:16.541 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:16.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fab9805c2a0 msgr2=0x7fab9805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:16.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fab9805c2a0 0x7fab9805e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7faba00144e0 tx=0x7faba0016040).stop 2021-11-20T09:00:16.543 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f5fc0 msgr2=0x7fabb0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:16.543 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f5fc0 0x7fabb0107350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7faba400e7f0 tx=0x7faba400b040).stop 2021-11-20T09:00:16.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 shutdown_connections 2021-11-20T09:00:16.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fab9805c2a0 0x7fab9805e760 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:16.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabb00f4d20 0x7fabb0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:16.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 --2- 172.21.15.93:0/2587439944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabb00f5fc0 0x7fabb0107350 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:16.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 >> 172.21.15.93:0/2587439944 conn(0x7fabb00f0440 msgr2=0x7fabb00f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:16.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 shutdown_connections 2021-11-20T09:00:16.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:16.541+0000 7fabb55c5700 1 -- 172.21.15.93:0/2587439944 wait complete. 2021-11-20T09:00:16.547 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:16.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:16.553 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:17.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:17 smithi102 conmon[36472]: cluster 2021-11-20T09:00:15.572913+0000 mgr.smithi093.qdikkm (mgr.14162) 291 : cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T09:00:17.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:17 smithi102 conmon[36472]: 2021-11-20T09:00:17.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:17 smithi102 conmon[36472]: audit 2021-11-20T09:00:16.541010+0000 mon.smithi093 (mon.0) 782 : audit [DBG] from='client.? 172.21.15.93:0/2587439944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:17.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:17 smithi093 conmon[33522]: cluster 2021-11-20T09:00:15.572913+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T09:00:17.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:17 smithi093 conmon[33522]: 14162) 291 : cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 3.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T09:00:17.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:17 smithi093 conmon[33522]: audit 2021-11-20T09:00:16.541010+0000 mon.smithi093 (mon.0) 782 : audit 2021-11-20T09:00:17.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:17 smithi093 conmon[33522]: [DBG] from='client.? 172.21.15.93:0/2587439944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:17.554 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:17.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 -- 172.21.15.93:0/4160670856 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 msgr2=0x7fda780fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:17.684 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 --2- 172.21.15.93:0/4160670856 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda780fbc50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fda60004660 tx=0x7fda60009b30).stop 2021-11-20T09:00:17.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 -- 172.21.15.93:0/4160670856 shutdown_connections 2021-11-20T09:00:17.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 --2- 172.21.15.93:0/4160670856 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda780fbc50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:17.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 --2- 172.21.15.93:0/4160670856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda78060100 0x7fda780f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:17.685 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 -- 172.21.15.93:0/4160670856 >> 172.21.15.93:0/4160670856 conn(0x7fda780f0ce0 msgr2=0x7fda780f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:17.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 -- 172.21.15.93:0/4160670856 shutdown_connections 2021-11-20T09:00:17.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 -- 172.21.15.93:0/4160670856 wait complete. 2021-11-20T09:00:17.686 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 Processor -- start 2021-11-20T09:00:17.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.680+0000 7fda7d87d700 1 -- start start 2021-11-20T09:00:17.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda7d87d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda78060100 0x7fda78106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:17.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda7d87d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda78107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:17.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda7d87d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda781079c0 con 0x7fda780f9840 2021-11-20T09:00:17.687 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda7d87d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda78107b00 con 0x7fda78060100 2021-11-20T09:00:17.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda767fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda78107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:17.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda767fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda78107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44216/0 (socket says 172.21.15.93:44216) 2021-11-20T09:00:17.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda767fc700 1 -- 172.21.15.93:0/3019635697 learned_addr learned my addr 172.21.15.93:0/3019635697 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:17.688 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda767fc700 1 -- 172.21.15.93:0/3019635697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda78060100 msgr2=0x7fda78106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:17.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda767fc700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda78060100 0x7fda78106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:17.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda767fc700 1 -- 172.21.15.93:0/3019635697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda60005040 con 0x7fda780f9840 2021-11-20T09:00:17.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda767fc700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda78107380 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fda60009f50 tx=0x7fda60009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:17.689 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda6f7fe700 1 -- 172.21.15.93:0/3019635697 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda60018070 con 0x7fda780f9840 2021-11-20T09:00:17.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda6f7fe700 1 -- 172.21.15.93:0/3019635697 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda60013410 con 0x7fda780f9840 2021-11-20T09:00:17.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7810c540 con 0x7fda780f9840 2021-11-20T09:00:17.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda6f7fe700 1 -- 172.21.15.93:0/3019635697 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda6001c620 con 0x7fda780f9840 2021-11-20T09:00:17.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.681+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7810caa0 con 0x7fda780f9840 2021-11-20T09:00:17.690 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.682+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda78027a00 con 0x7fda780f9840 2021-11-20T09:00:17.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.683+0000 7fda6f7fe700 1 -- 172.21.15.93:0/3019635697 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fda6001d420 con 0x7fda780f9840 2021-11-20T09:00:17.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.683+0000 7fda6f7fe700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fda6405c2a0 0x7fda6405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:17.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.683+0000 7fda6f7fe700 1 -- 172.21.15.93:0/3019635697 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fda600781f0 con 0x7fda780f9840 2021-11-20T09:00:17.691 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.683+0000 7fda76ffd700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fda6405c2a0 0x7fda6405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:17.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.683+0000 7fda76ffd700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fda6405c2a0 0x7fda6405e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fda68006fd0 tx=0x7fda68006e30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:17.692 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.687+0000 7fda6f7fe700 1 -- 172.21.15.93:0/3019635697 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda6004aa50 con 0x7fda780f9840 2021-11-20T09:00:17.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.869+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda780feeb0 con 0x7fda780f9840 2021-11-20T09:00:17.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.869+0000 7fda6f7fe700 1 -- 172.21.15.93:0/3019635697 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7fda6004e070 con 0x7fda780f9840 2021-11-20T09:00:17.870 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:17.870 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:17.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fda6405c2a0 msgr2=0x7fda6405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:17.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fda6405c2a0 0x7fda6405e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fda68006fd0 tx=0x7fda68006e30).stop 2021-11-20T09:00:17.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 msgr2=0x7fda78107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:17.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda78107380 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fda60009f50 tx=0x7fda60009b30).stop 2021-11-20T09:00:17.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 shutdown_connections 2021-11-20T09:00:17.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fda6405c2a0 0x7fda6405e760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:17.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda78060100 0x7fda78106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:17.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 --2- 172.21.15.93:0/3019635697 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda780f9840 0x7fda78107380 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:17.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 >> 172.21.15.93:0/3019635697 conn(0x7fda780f0ce0 msgr2=0x7fda780f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:17.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 shutdown_connections 2021-11-20T09:00:17.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:17.871+0000 7fda7d87d700 1 -- 172.21.15.93:0/3019635697 wait complete. 2021-11-20T09:00:17.874 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:17.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:17.885 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:18.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:18 smithi093 conmon[33522]: audit 2021-11-20T09:00:17.871527+0000 mon.smithi093 (mon.0) 783 2021-11-20T09:00:18.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:18 smithi093 conmon[33522]: : audit [DBG] from='client.? 172.21.15.93:0/3019635697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:18.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:18 smithi102 conmon[36472]: audit 2021-11-20T09:00:17.871527+0000 mon.smithi093 (mon.0) 783 : audit [DBG] from='client.? 172.21.15.93:0/3019635697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:18.886 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:19.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.012+0000 7f49ab1e2700 1 -- 172.21.15.93:0/749741261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 msgr2=0x7f49a40f7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:19.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.012+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/749741261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a40f7960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f4994004660 tx=0x7f4994009b30).stop 2021-11-20T09:00:19.034 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 -- 172.21.15.93:0/749741261 shutdown_connections 2021-11-20T09:00:19.034 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/749741261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a40f7960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:19.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/749741261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49a40f4840 0x7f49a40f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:19.035 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 -- 172.21.15.93:0/749741261 >> 172.21.15.93:0/749741261 conn(0x7f49a40f0360 msgr2=0x7f49a40f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:19.036 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 -- 172.21.15.93:0/749741261 shutdown_connections 2021-11-20T09:00:19.036 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 -- 172.21.15.93:0/749741261 wait complete. 2021-11-20T09:00:19.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 Processor -- start 2021-11-20T09:00:19.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 -- start start 2021-11-20T09:00:19.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49a40f4840 0x7f49a4106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:19.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a41072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:19.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a41078b0 con 0x7f49a40f51e0 2021-11-20T09:00:19.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49ab1e2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a41080f0 con 0x7f49a40f4840 2021-11-20T09:00:19.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49a99df700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a41072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:19.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49a99df700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a41072b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44226/0 (socket says 172.21.15.93:44226) 2021-11-20T09:00:19.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.013+0000 7f49a99df700 1 -- 172.21.15.93:0/2845867195 learned_addr learned my addr 172.21.15.93:0/2845867195 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:19.041 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f49aa1e0700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49a40f4840 0x7f49a4106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:19.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f49a99df700 1 -- 172.21.15.93:0/2845867195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49a40f4840 msgr2=0x7f49a4106dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:19.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f49a99df700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49a40f4840 0x7f49a4106dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:19.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f49a99df700 1 -- 172.21.15.93:0/2845867195 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994005040 con 0x7f49a40f51e0 2021-11-20T09:00:19.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f49a99df700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a41072b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f4994009f50 tx=0x7f4994009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:19.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f499affd700 1 -- 172.21.15.93:0/2845867195 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4994018070 con 0x7f49a40f51e0 2021-11-20T09:00:19.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f499affd700 1 -- 172.21.15.93:0/2845867195 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4994013410 con 0x7f49a40f51e0 2021-11-20T09:00:19.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a4108370 con 0x7f49a40f51e0 2021-11-20T09:00:19.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f499affd700 1 -- 172.21.15.93:0/2845867195 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499401c3c0 con 0x7f49a40f51e0 2021-11-20T09:00:19.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.014+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a4108850 con 0x7f49a40f51e0 2021-11-20T09:00:19.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.015+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a40f3ff0 con 0x7f49a40f51e0 2021-11-20T09:00:19.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.015+0000 7f499affd700 1 -- 172.21.15.93:0/2845867195 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f499401d420 con 0x7f49a40f51e0 2021-11-20T09:00:19.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.016+0000 7f499affd700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f499005c300 0x7f499005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:19.048 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.016+0000 7f499affd700 1 -- 172.21.15.93:0/2845867195 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f4994077f30 con 0x7f49a40f51e0 2021-11-20T09:00:19.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.016+0000 7f49aa1e0700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f499005c300 0x7f499005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:19.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.016+0000 7f49aa1e0700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f499005c300 0x7f499005e7c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f49a4107b80 tx=0x7f49a0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:19.049 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.020+0000 7f499affd700 1 -- 172.21.15.93:0/2845867195 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49940a42b0 con 0x7f49a40f51e0 2021-11-20T09:00:19.204 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.204+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a4045ad0 con 0x7f49a40f51e0 2021-11-20T09:00:19.204 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.205+0000 7f499affd700 1 -- 172.21.15.93:0/2845867195 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f499404a870 con 0x7f49a40f51e0 2021-11-20T09:00:19.205 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:19.205 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:19.206 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:19 smithi093 conmon[33522]: cluster 2021-11-20T09:00:17.573275+0000 mgr.smithi093.qdikkm (mgr.14162) 292 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:19.208 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f499005c300 msgr2=0x7f499005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:19.208 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f499005c300 0x7f499005e7c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f49a4107b80 tx=0x7f49a0009040).stop 2021-11-20T09:00:19.208 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 msgr2=0x7f49a41072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:19.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a41072b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f4994009f50 tx=0x7f4994009b30).stop 2021-11-20T09:00:19.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 shutdown_connections 2021-11-20T09:00:19.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f499005c300 0x7f499005e7c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:19.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49a40f4840 0x7f49a4106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:19.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 --2- 172.21.15.93:0/2845867195 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49a40f51e0 0x7f49a41072b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:19.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 >> 172.21.15.93:0/2845867195 conn(0x7f49a40f0360 msgr2=0x7f49a40fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:19.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 shutdown_connections 2021-11-20T09:00:19.211 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:19.207+0000 7f49ab1e2700 1 -- 172.21.15.93:0/2845867195 wait complete. 2021-11-20T09:00:19.212 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:19.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:19.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:19.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:19 smithi102 conmon[36472]: cluster 2021-11-20T09:00:17.573275+0000 mgr.smithi093.qdikkm (mgr.14162) 292 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:20.225 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:20.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 -- 172.21.15.93:0/3233848220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80f9880 msgr2=0x7fb8f80fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:20.351 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/3233848220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80f9880 0x7fb8f80fbc70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fb8e0004660 tx=0x7fb8e0009b30).stop 2021-11-20T09:00:20.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 -- 172.21.15.93:0/3233848220 shutdown_connections 2021-11-20T09:00:20.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/3233848220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80f9880 0x7fb8f80fbc70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:20.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/3233848220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8f80600c0 0x7fb8f80f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:20.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 -- 172.21.15.93:0/3233848220 >> 172.21.15.93:0/3233848220 conn(0x7fb8f80f0ce0 msgr2=0x7fb8f80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:20.352 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 -- 172.21.15.93:0/3233848220 shutdown_connections 2021-11-20T09:00:20.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.349+0000 7fb8fddbc700 1 -- 172.21.15.93:0/3233848220 wait complete. 2021-11-20T09:00:20.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8fddbc700 1 Processor -- start 2021-11-20T09:00:20.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8fddbc700 1 -- start start 2021-11-20T09:00:20.353 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8fddbc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80600c0 0x7fb8f8106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:20.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8fddbc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8f80f9880 0x7fb8f8107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:20.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8fddbc700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8107940 con 0x7fb8f80600c0 2021-11-20T09:00:20.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8fddbc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8107a80 con 0x7fb8f80f9880 2021-11-20T09:00:20.354 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f77fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80600c0 0x7fb8f8106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:20.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f77fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80600c0 0x7fb8f8106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44236/0 (socket says 172.21.15.93:44236) 2021-11-20T09:00:20.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f77fe700 1 -- 172.21.15.93:0/2010299674 learned_addr learned my addr 172.21.15.93:0/2010299674 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:20.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f77fe700 1 -- 172.21.15.93:0/2010299674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8f80f9880 msgr2=0x7fb8f8107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:20.355 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f77fe700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8f80f9880 0x7fb8f8107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:20.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f77fe700 1 -- 172.21.15.93:0/2010299674 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0005040 con 0x7fb8f80600c0 2021-11-20T09:00:20.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f77fe700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80600c0 0x7fb8f8106ea0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb8e800c530 tx=0x7fb8e8007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:20.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.350+0000 7fb8f4ff9700 1 -- 172.21.15.93:0/2010299674 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e800f650 con 0x7fb8f80600c0 2021-11-20T09:00:20.356 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.351+0000 7fb8f4ff9700 1 -- 172.21.15.93:0/2010299674 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8e800fc60 con 0x7fb8f80600c0 2021-11-20T09:00:20.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.351+0000 7fb8f4ff9700 1 -- 172.21.15.93:0/2010299674 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e8012d30 con 0x7fb8f80600c0 2021-11-20T09:00:20.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.351+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f810c520 con 0x7fb8f80600c0 2021-11-20T09:00:20.357 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.351+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f810ca90 con 0x7fb8f80600c0 2021-11-20T09:00:20.358 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.351+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f8027a00 con 0x7fb8f80600c0 2021-11-20T09:00:20.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.352+0000 7fb8f4ff9700 1 -- 172.21.15.93:0/2010299674 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb8e8010070 con 0x7fb8f80600c0 2021-11-20T09:00:20.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.352+0000 7fb8f4ff9700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb8e405c2a0 0x7fb8e405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:20.359 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.352+0000 7fb8f4ff9700 1 -- 172.21.15.93:0/2010299674 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb8e8076050 con 0x7fb8f80600c0 2021-11-20T09:00:20.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.352+0000 7fb8eeffd700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb8e405c2a0 0x7fb8e405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:20.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.353+0000 7fb8eeffd700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb8e405c2a0 0x7fb8e405e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb8e0009f50 tx=0x7fb8e0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:20.360 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.356+0000 7fb8f4ff9700 1 -- 172.21.15.93:0/2010299674 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8e80452b0 con 0x7fb8f80600c0 2021-11-20T09:00:20.417 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:20 smithi102 conmon[36472]: audit 2021-11-20T09:00:19.206542+0000 mon.smithi093 (mon.0) 784 : audit [DBG] from='client.? 172.21.15.93:0/2845867195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:20.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:20 smithi093 conmon[33522]: audit 2021-11-20T09:00:19.206542+0000 mon.smithi093 (mon.0) 784 : audit [DBG] from='client.? 172.21.15.93:0/2845867195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:20.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:20 smithi093 conmon[33522]: 2021-11-20T09:00:20.541 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.541+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8f80fee30 con 0x7fb8f80600c0 2021-11-20T09:00:20.542 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.542+0000 7fb8f4ff9700 1 -- 172.21.15.93:0/2010299674 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7fb8e8048910 con 0x7fb8f80600c0 2021-11-20T09:00:20.542 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:20.543 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:20.544 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb8e405c2a0 msgr2=0x7fb8e405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:20.544 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb8e405c2a0 0x7fb8e405e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb8e0009f50 tx=0x7fb8e0009b30).stop 2021-11-20T09:00:20.544 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80600c0 msgr2=0x7fb8f8106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:20.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80600c0 0x7fb8f8106ea0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb8e800c530 tx=0x7fb8e8007e20).stop 2021-11-20T09:00:20.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 shutdown_connections 2021-11-20T09:00:20.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb8e405c2a0 0x7fb8e405e760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:20.545 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8f80600c0 0x7fb8f8106ea0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:20.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 --2- 172.21.15.93:0/2010299674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8f80f9880 0x7fb8f8107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:20.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 >> 172.21.15.93:0/2010299674 conn(0x7fb8f80f0ce0 msgr2=0x7fb8f80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:20.546 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 shutdown_connections 2021-11-20T09:00:20.547 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:20.543+0000 7fb8fddbc700 1 -- 172.21.15.93:0/2010299674 wait complete. 2021-11-20T09:00:20.547 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:20.555 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:20.556 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:21.418 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:21 smithi102 conmon[36472]: cluster 2021-11-20T09:00:19.573692+0000 mgr.smithi093.qdikkm (mgr.14162) 293 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:21.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:21 smithi102 conmon[36472]: audit 2021 2021-11-20T09:00:21.422 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:21 smithi102 conmon[36472]: -11-20T09:00:20.543653+0000 mon.smithi093 (mon.0) 785 : audit [DBG] from='client.? 172.21.15.93:0/2010299674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:21.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:21 smithi093 conmon[33522]: cluster 2021-11-20T09:00:19.573692+0000 2021-11-20T09:00:21.461 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:21 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 293 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:21.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:21 smithi093 conmon[33522]: audit 2021-11-20T09:00:20.543653+0000 mon.smithi093 ( 2021-11-20T09:00:21.462 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:21 smithi093 conmon[33522]: mon.0) 785 : audit [DBG] from='client.? 172.21.15.93:0/2010299674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:21.558 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:21.704 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 -- 172.21.15.93:0/212812980 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f7950 msgr2=0x7f62a00f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:21.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 --2- 172.21.15.93:0/212812980 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f7950 0x7f62a00f7db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f6294004660 tx=0x7f6294009b30).stop 2021-11-20T09:00:21.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 -- 172.21.15.93:0/212812980 shutdown_connections 2021-11-20T09:00:21.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 --2- 172.21.15.93:0/212812980 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f7950 0x7f62a00f7db0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:21.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 --2- 172.21.15.93:0/212812980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62a00f66b0 0x7f62a00f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:21.705 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 -- 172.21.15.93:0/212812980 >> 172.21.15.93:0/212812980 conn(0x7f62a00f1e10 msgr2=0x7f62a00f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:21.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 -- 172.21.15.93:0/212812980 shutdown_connections 2021-11-20T09:00:21.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 -- 172.21.15.93:0/212812980 wait complete. 2021-11-20T09:00:21.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.702+0000 7f62a4e47700 1 Processor -- start 2021-11-20T09:00:21.706 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f62a4e47700 1 -- start start 2021-11-20T09:00:21.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f62a4e47700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f66b0 0x7f62a0106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:21.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f62a4e47700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62a00f7950 0x7f62a01072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:21.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f62a4e47700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0107870 con 0x7f62a00f66b0 2021-11-20T09:00:21.707 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f62a4e47700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01079b0 con 0x7f62a00f7950 2021-11-20T09:00:21.708 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f66b0 0x7f62a0106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:21.708 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f66b0 0x7f62a0106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44246/0 (socket says 172.21.15.93:44246) 2021-11-20T09:00:21.708 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629e59c700 1 -- 172.21.15.93:0/3788444499 learned_addr learned my addr 172.21.15.93:0/3788444499 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:21.708 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629dd9b700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62a00f7950 0x7f62a01072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:21.708 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629e59c700 1 -- 172.21.15.93:0/3788444499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62a00f7950 msgr2=0x7f62a01072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:21.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629e59c700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62a00f7950 0x7f62a01072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:21.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629e59c700 1 -- 172.21.15.93:0/3788444499 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294005040 con 0x7f62a00f66b0 2021-11-20T09:00:21.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f629e59c700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f66b0 0x7f62a0106dd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f629000d8d0 tx=0x7f6290004f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:21.709 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f628effd700 1 -- 172.21.15.93:0/3788444499 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f629000de70 con 0x7f62a00f66b0 2021-11-20T09:00:21.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f628effd700 1 -- 172.21.15.93:0/3788444499 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f629000f040 con 0x7f62a00f66b0 2021-11-20T09:00:21.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a010c450 con 0x7f62a00f66b0 2021-11-20T09:00:21.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.703+0000 7f628effd700 1 -- 172.21.15.93:0/3788444499 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6290013a90 con 0x7f62a00f66b0 2021-11-20T09:00:21.710 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.704+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a010ca10 con 0x7f62a00f66b0 2021-11-20T09:00:21.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.704+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a0027a00 con 0x7f62a00f66b0 2021-11-20T09:00:21.711 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.705+0000 7f628effd700 1 -- 172.21.15.93:0/3788444499 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f629001a030 con 0x7f62a00f66b0 2021-11-20T09:00:21.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.705+0000 7f628effd700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f628805c2a0 0x7f628805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:21.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.705+0000 7f628effd700 1 -- 172.21.15.93:0/3788444499 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f6290076200 con 0x7f62a00f66b0 2021-11-20T09:00:21.712 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.705+0000 7f629dd9b700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f628805c2a0 0x7f628805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:21.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.706+0000 7f629dd9b700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f628805c2a0 0x7f628805e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6294009d60 tx=0x7f6294009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:21.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.709+0000 7f628effd700 1 -- 172.21.15.93:0/3788444499 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6290048ac0 con 0x7f62a00f66b0 2021-11-20T09:00:21.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.896+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62a00fee40 con 0x7f62a00f66b0 2021-11-20T09:00:21.896 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.896+0000 7f628effd700 1 -- 172.21.15.93:0/3788444499 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f6290018090 con 0x7f62a00f66b0 2021-11-20T09:00:21.897 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:21.897 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:21.898 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f628805c2a0 msgr2=0x7f628805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:21.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f628805c2a0 0x7f628805e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6294009d60 tx=0x7f6294009b30).stop 2021-11-20T09:00:21.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f66b0 msgr2=0x7f62a0106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:21.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f66b0 0x7f62a0106dd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f629000d8d0 tx=0x7f6290004f00).stop 2021-11-20T09:00:21.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 shutdown_connections 2021-11-20T09:00:21.899 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f628805c2a0 0x7f628805e760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:21.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62a00f66b0 0x7f62a0106dd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:21.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 --2- 172.21.15.93:0/3788444499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62a00f7950 0x7f62a01072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:21.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 >> 172.21.15.93:0/3788444499 conn(0x7f62a00f1e10 msgr2=0x7f62a00fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:21.900 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 shutdown_connections 2021-11-20T09:00:21.901 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:21.898+0000 7f62a4e47700 1 -- 172.21.15.93:0/3788444499 wait complete. 2021-11-20T09:00:21.901 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:21.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:21.910 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:22.911 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:22.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:22 smithi102 conmon[36472]: cluster 2021-11-20T09:00:21.574166+0000 mgr.smithi093.qdikkm (mgr.14162) 294 : cluster [DBG] pgmap v269: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:22.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:22 smithi102 conmon[36472]: audit 2021-11-20T09:00:21.898370+0000 mon.smithi093 (mon.0) 786 : audit [DBG] from='client.? 172.21.15.93:0/3788444499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:22.937 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:22 smithi093 conmon[33522]: cluster 2021-11-20T09:00:21.574166+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T09:00:22.937 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:22 smithi093 conmon[33522]: .14162) 294 : cluster [DBG] pgmap v269: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:22.938 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:22 smithi093 conmon[33522]: audit 2021-11-20T09:00:21.898370+0000 mon.smithi093 (mon.0) 786 : audit 2021-11-20T09:00:22.938 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:22 smithi093 conmon[33522]: [DBG] from='client.? 172.21.15.93:0/3788444499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:23.037 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 -- 172.21.15.93:0/1977029841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f6290 msgr2=0x7f6adc0f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:23.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1977029841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f6290 0x7f6adc0f66f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6ad0004660 tx=0x7f6ad0009b30).stop 2021-11-20T09:00:23.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 -- 172.21.15.93:0/1977029841 shutdown_connections 2021-11-20T09:00:23.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1977029841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f6290 0x7f6adc0f66f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:23.038 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1977029841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6adc0f7c80 0x7f6adc0f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:23.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 -- 172.21.15.93:0/1977029841 >> 172.21.15.93:0/1977029841 conn(0x7f6adc0f1d10 msgr2=0x7f6adc0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:23.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 -- 172.21.15.93:0/1977029841 shutdown_connections 2021-11-20T09:00:23.039 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.035+0000 7f6ae2748700 1 -- 172.21.15.93:0/1977029841 wait complete. 2021-11-20T09:00:23.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae2748700 1 Processor -- start 2021-11-20T09:00:23.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae2748700 1 -- start start 2021-11-20T09:00:23.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae2748700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6adc0f6290 0x7f6adc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:23.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae2748700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f7c80 0x7f6adc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:23.040 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae2748700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc06a410 con 0x7f6adc0f7c80 2021-11-20T09:00:23.041 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae2748700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc06a550 con 0x7f6adc0f6290 2021-11-20T09:00:23.041 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae0f45700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f7c80 0x7f6adc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:23.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae0f45700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f7c80 0x7f6adc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44258/0 (socket says 172.21.15.93:44258) 2021-11-20T09:00:23.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae0f45700 1 -- 172.21.15.93:0/1570206434 learned_addr learned my addr 172.21.15.93:0/1570206434 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:23.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae0f45700 1 -- 172.21.15.93:0/1570206434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6adc0f6290 msgr2=0x7f6adc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:23.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae1746700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6adc0f6290 0x7f6adc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:23.042 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae0f45700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6adc0f6290 0x7f6adc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:23.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.036+0000 7f6ae0f45700 1 -- 172.21.15.93:0/1570206434 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0005040 con 0x7f6adc0f7c80 2021-11-20T09:00:23.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.037+0000 7f6ae0f45700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f7c80 0x7f6adc069e20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f6ad0009830 tx=0x7f6ad0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:23.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.037+0000 7f6ad9ffb700 1 -- 172.21.15.93:0/1570206434 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad0018070 con 0x7f6adc0f7c80 2021-11-20T09:00:23.043 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.037+0000 7f6ad9ffb700 1 -- 172.21.15.93:0/1570206434 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ad0013410 con 0x7f6adc0f7c80 2021-11-20T09:00:23.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.037+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc06a7d0 con 0x7f6adc0f7c80 2021-11-20T09:00:23.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.037+0000 7f6ad9ffb700 1 -- 172.21.15.93:0/1570206434 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad001c620 con 0x7f6adc0f7c80 2021-11-20T09:00:23.044 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.037+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc10ee80 con 0x7f6adc0f7c80 2021-11-20T09:00:23.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.038+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc045ad0 con 0x7f6adc0f7c80 2021-11-20T09:00:23.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.038+0000 7f6ad9ffb700 1 -- 172.21.15.93:0/1570206434 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6ad001c8e0 con 0x7f6adc0f7c80 2021-11-20T09:00:23.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.039+0000 7f6ad9ffb700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6ac405c300 0x7f6ac405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:23.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.039+0000 7f6ad9ffb700 1 -- 172.21.15.93:0/1570206434 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f6ad00781a0 con 0x7f6adc0f7c80 2021-11-20T09:00:23.045 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.039+0000 7f6ae1746700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6ac405c300 0x7f6ac405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:23.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.039+0000 7f6ae1746700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6ac405c300 0x7f6ac405e7c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6adc06b730 tx=0x7f6acc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:23.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.043+0000 7f6ad9ffb700 1 -- 172.21.15.93:0/1570206434 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ad004aa00 con 0x7f6adc0f7c80 2021-11-20T09:00:23.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.227+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6adc027a00 con 0x7f6adc0f7c80 2021-11-20T09:00:23.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.227+0000 7f6ad9ffb700 1 -- 172.21.15.93:0/1570206434 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f6ad004e020 con 0x7f6adc0f7c80 2021-11-20T09:00:23.229 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:23.229 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:23.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6ac405c300 msgr2=0x7f6ac405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:23.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6ac405c300 0x7f6ac405e7c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6adc06b730 tx=0x7f6acc008040).stop 2021-11-20T09:00:23.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f7c80 msgr2=0x7f6adc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:23.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f7c80 0x7f6adc069e20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f6ad0009830 tx=0x7f6ad0009b30).stop 2021-11-20T09:00:23.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 shutdown_connections 2021-11-20T09:00:23.235 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6ac405c300 0x7f6ac405e7c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:23.235 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6adc0f6290 0x7f6adc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:23.235 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 --2- 172.21.15.93:0/1570206434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6adc0f7c80 0x7f6adc069e20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:23.236 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 >> 172.21.15.93:0/1570206434 conn(0x7f6adc0f1d10 msgr2=0x7f6adc0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:23.236 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 shutdown_connections 2021-11-20T09:00:23.236 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:23.229+0000 7f6ae2748700 1 -- 172.21.15.93:0/1570206434 wait complete. 2021-11-20T09:00:23.237 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:23.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:23.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:23.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:23 smithi102 conmon[36472]: audit 2021-11-20T09:00:23.228192+0000 mon.smithi093 (mon.0 2021-11-20T09:00:23.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:23 smithi102 conmon[36472]: ) 787 : audit [DBG] from='client.? 172.21.15.93:0/1570206434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:23.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:23 smithi093 conmon[33522]: audit 2021-11-20T09:00:23.228192+0000 mon.smithi093 (mon. 2021-11-20T09:00:23.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:23 smithi093 conmon[33522]: 0) 787 : audit [DBG] from='client.? 172.21.15.93:0/1570206434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:24.253 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:24.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/3635877233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f6c70 msgr2=0x7f0dd00f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:24.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/3635877233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f6c70 0x7f0dd00f7090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0dc0004660 tx=0x7f0dc0009b30).stop 2021-11-20T09:00:24.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/3635877233 shutdown_connections 2021-11-20T09:00:24.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/3635877233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dd00f5970 0x7f0dd00f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:24.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/3635877233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f6c70 0x7f0dd00f7090 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:24.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/3635877233 >> 172.21.15.93:0/3635877233 conn(0x7f0dd00f04a0 msgr2=0x7f0dd00f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:24.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/3635877233 shutdown_connections 2021-11-20T09:00:24.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.383+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/3635877233 wait complete. 2021-11-20T09:00:24.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dd4eb6700 1 Processor -- start 2021-11-20T09:00:24.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dd4eb6700 1 -- start start 2021-11-20T09:00:24.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dd4eb6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f5970 0x7f0dd00f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:24.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dd4eb6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dd00f6c70 0x7f0dd010bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:24.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dd4eb6700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd010c270 con 0x7f0dd00f5970 2021-11-20T09:00:24.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dd4eb6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd010c3b0 con 0x7f0dd00f6c70 2021-11-20T09:00:24.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dce59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f5970 0x7f0dd00f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:24.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dce59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f5970 0x7f0dd00f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44268/0 (socket says 172.21.15.93:44268) 2021-11-20T09:00:24.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dce59c700 1 -- 172.21.15.93:0/1698428406 learned_addr learned my addr 172.21.15.93:0/1698428406 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:24.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dcdd9b700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dd00f6c70 0x7f0dd010bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:24.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.384+0000 7f0dce59c700 1 -- 172.21.15.93:0/1698428406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dd00f6c70 msgr2=0x7f0dd010bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:24.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dce59c700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dd00f6c70 0x7f0dd010bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:24.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dce59c700 1 -- 172.21.15.93:0/1698428406 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0005040 con 0x7f0dd00f5970 2021-11-20T09:00:24.392 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dce59c700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f5970 0x7f0dd00f5310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f0dc0004e30 tx=0x7f0dc0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:24.392 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dbeffd700 1 -- 172.21.15.93:0/1698428406 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dc0018070 con 0x7f0dd00f5970 2021-11-20T09:00:24.392 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dbeffd700 1 -- 172.21.15.93:0/1698428406 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dc0013410 con 0x7f0dd00f5970 2021-11-20T09:00:24.392 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd010c5b0 con 0x7f0dd00f5970 2021-11-20T09:00:24.393 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dbeffd700 1 -- 172.21.15.93:0/1698428406 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dc001c3c0 con 0x7f0dd00f5970 2021-11-20T09:00:24.393 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.385+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd010cae0 con 0x7f0dd00f5970 2021-11-20T09:00:24.393 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.386+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd0027a00 con 0x7f0dd00f5970 2021-11-20T09:00:24.393 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.386+0000 7f0dbeffd700 1 -- 172.21.15.93:0/1698428406 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0dc001d420 con 0x7f0dd00f5970 2021-11-20T09:00:24.394 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.386+0000 7f0dbeffd700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0db805c2a0 0x7f0db805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:24.394 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.387+0000 7f0dbeffd700 1 -- 172.21.15.93:0/1698428406 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f0dc0077f60 con 0x7f0dd00f5970 2021-11-20T09:00:24.394 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.387+0000 7f0dcdd9b700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0db805c2a0 0x7f0db805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:24.395 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.387+0000 7f0dcdd9b700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0db805c2a0 0x7f0db805e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0dc40096c0 tx=0x7f0dc4009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:24.395 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.390+0000 7f0dbeffd700 1 -- 172.21.15.93:0/1698428406 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0dc00a42b0 con 0x7f0dd00f5970 2021-11-20T09:00:24.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.573+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dd00fde20 con 0x7f0dd00f5970 2021-11-20T09:00:24.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.573+0000 7f0dbeffd700 1 -- 172.21.15.93:0/1698428406 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f0dc004a820 con 0x7f0dd00f5970 2021-11-20T09:00:24.573 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:24.574 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:24.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0db805c2a0 msgr2=0x7f0db805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:24.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0db805c2a0 0x7f0db805e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0dc40096c0 tx=0x7f0dc4009be0).stop 2021-11-20T09:00:24.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f5970 msgr2=0x7f0dd00f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:24.576 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f5970 0x7f0dd00f5310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f0dc0004e30 tx=0x7f0dc0009b30).stop 2021-11-20T09:00:24.576 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 shutdown_connections 2021-11-20T09:00:24.576 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f0db805c2a0 0x7f0db805e760 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:24.576 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0dd00f5970 0x7f0dd00f5310 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:24.577 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 --2- 172.21.15.93:0/1698428406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dd00f6c70 0x7f0dd010bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:24.577 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 >> 172.21.15.93:0/1698428406 conn(0x7f0dd00f04a0 msgr2=0x7f0dd00fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:24.577 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 shutdown_connections 2021-11-20T09:00:24.577 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:24.575+0000 7f0dd4eb6700 1 -- 172.21.15.93:0/1698428406 wait complete. 2021-11-20T09:00:24.577 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:24.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:24.587 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:24.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:24 smithi102 conmon[36472]: cluster 2021-11-20T09:00:23.574662+0000 mgr.smithi093.qdikkm (mgr.14162) 295 : cluster 2021-11-20T09:00:24.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:24 smithi102 conmon[36472]: [DBG] pgmap v270: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:24.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:24 smithi102 conmon[36472]: audit 2021-11-20T09:00:24.574404+0000 mon.smithi093 (mon.0) 788 : audit [DBG] from='client.? 172.21.15.93:0/1698428406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:24.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:24 smithi093 conmon[33522]: cluster 2021-11-20T09:00:23.574662+0000 mgr.smithi093.qdikkm (mgr.14162) 295 : cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:24.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:24 smithi093 conmon[33522]: audit 2021-11-20T09:00:24.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:24 smithi093 conmon[33522]: 2021-11-20T09:00:24.574404+0000 mon.smithi093 (mon.0) 788 : audit [DBG] from='client.? 172.21.15.93:0/1698428406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:25.588 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:25.715 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 -- 172.21.15.93:0/819809555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f51e0 msgr2=0x7f9d000f7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:25.715 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 --2- 172.21.15.93:0/819809555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f51e0 0x7f9d000f7960 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f9cf0004660 tx=0x7f9cf0009b30).stop 2021-11-20T09:00:25.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 -- 172.21.15.93:0/819809555 shutdown_connections 2021-11-20T09:00:25.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 --2- 172.21.15.93:0/819809555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f51e0 0x7f9d000f7960 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:25.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 --2- 172.21.15.93:0/819809555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d000f4840 0x7f9d000f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:25.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 -- 172.21.15.93:0/819809555 >> 172.21.15.93:0/819809555 conn(0x7f9d000f0360 msgr2=0x7f9d000f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:25.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 -- 172.21.15.93:0/819809555 shutdown_connections 2021-11-20T09:00:25.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.715+0000 7f9d049a3700 1 -- 172.21.15.93:0/819809555 wait complete. 2021-11-20T09:00:25.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9d049a3700 1 Processor -- start 2021-11-20T09:00:25.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9d049a3700 1 -- start start 2021-11-20T09:00:25.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9d049a3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f4840 0x7f9d00106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:25.722 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9d049a3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d000f51e0 0x7f9d001072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:25.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9d049a3700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d001078b0 con 0x7f9d000f4840 2021-11-20T09:00:25.723 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9d049a3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d001080f0 con 0x7f9d000f51e0 2021-11-20T09:00:25.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9cff7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f4840 0x7f9d00106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:25.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9cff7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f4840 0x7f9d00106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44278/0 (socket says 172.21.15.93:44278) 2021-11-20T09:00:25.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9cff7fe700 1 -- 172.21.15.93:0/2981327053 learned_addr learned my addr 172.21.15.93:0/2981327053 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:25.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.716+0000 7f9cfeffd700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d000f51e0 0x7f9d001072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:25.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9cff7fe700 1 -- 172.21.15.93:0/2981327053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d000f51e0 msgr2=0x7f9d001072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:25.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9cff7fe700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d000f51e0 0x7f9d001072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:25.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9cff7fe700 1 -- 172.21.15.93:0/2981327053 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0005040 con 0x7f9d000f4840 2021-11-20T09:00:25.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9cff7fe700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f4840 0x7f9d00106dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f9cf4002700 tx=0x7f9cf4009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:25.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9ce7fff700 1 -- 172.21.15.93:0/2981327053 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cf40079c0 con 0x7f9d000f4840 2021-11-20T09:00:25.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9ce7fff700 1 -- 172.21.15.93:0/2981327053 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cf400f040 con 0x7f9d000f4840 2021-11-20T09:00:25.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d001083d0 con 0x7f9d000f4840 2021-11-20T09:00:25.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9ce7fff700 1 -- 172.21.15.93:0/2981327053 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cf4014860 con 0x7f9d000f4840 2021-11-20T09:00:25.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.717+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d00108910 con 0x7f9d000f4840 2021-11-20T09:00:25.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.718+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d000f3ff0 con 0x7f9d000f4840 2021-11-20T09:00:25.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.718+0000 7f9ce7fff700 1 -- 172.21.15.93:0/2981327053 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9cf401b060 con 0x7f9d000f4840 2021-11-20T09:00:25.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.719+0000 7f9ce7fff700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ce805c300 0x7f9ce805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:25.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.719+0000 7f9ce7fff700 1 -- 172.21.15.93:0/2981327053 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f9cf4076110 con 0x7f9d000f4840 2021-11-20T09:00:25.729 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.719+0000 7f9cfeffd700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ce805c300 0x7f9ce805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:25.729 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.719+0000 7f9cfeffd700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ce805c300 0x7f9ce805e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9d00106b50 tx=0x7f9cf0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:25.729 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.722+0000 7f9ce7fff700 1 -- 172.21.15.93:0/2981327053 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cf40489d0 con 0x7f9d000f4840 2021-11-20T09:00:25.910 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.906+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d00027b50 con 0x7f9d000f4840 2021-11-20T09:00:25.910 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.907+0000 7f9ce7fff700 1 -- 172.21.15.93:0/2981327053 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f9cf404bff0 con 0x7f9d000f4840 2021-11-20T09:00:25.911 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:25.911 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:25.914 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ce805c300 msgr2=0x7f9ce805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:25.915 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ce805c300 0x7f9ce805e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9d00106b50 tx=0x7f9cf0009b30).stop 2021-11-20T09:00:25.915 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f4840 msgr2=0x7f9d00106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:25.915 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f4840 0x7f9d00106dc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f9cf4002700 tx=0x7f9cf4009d80).stop 2021-11-20T09:00:25.915 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 shutdown_connections 2021-11-20T09:00:25.916 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f9ce805c300 0x7f9ce805e7c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:25.916 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9d000f4840 0x7f9d00106dc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:25.916 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 --2- 172.21.15.93:0/2981327053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d000f51e0 0x7f9d001072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:25.916 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 >> 172.21.15.93:0/2981327053 conn(0x7f9d000f0360 msgr2=0x7f9d000fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:25.916 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 shutdown_connections 2021-11-20T09:00:25.917 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:25.909+0000 7f9d049a3700 1 -- 172.21.15.93:0/2981327053 wait complete. 2021-11-20T09:00:25.917 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:25.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:25.924 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:26.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:26 smithi102 conmon[36472]: cluster 2021-11-20T09:00:25.575132+0000 mgr.smithi093.qdikkm (mgr.14162) 296 : cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:26.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:26 smithi102 conmon[36472]: audit 2021-11-20T09:00:25.907974+0000 2021-11-20T09:00:26.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:26 smithi102 conmon[36472]: mon.smithi093 (mon.0) 789 : audit [DBG] from='client.? 172.21.15.93:0/2981327053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:26.926 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:26.951 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:26 smithi093 conmon[33522]: cluster 2021-11-20T09:00:25.575132+0000 mgr.smithi093.qdikkm (mgr.14162) 296 : 2021-11-20T09:00:26.952 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:26 smithi093 conmon[33522]: cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:26.953 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:26 smithi093 conmon[33522]: audit 2021-11-20T09:00:25.907974+0000 mon.smithi093 (mon.0) 789 : audit [DBG] from='client.? 172.21.15.93:0/2981327053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:27.051 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- 172.21.15.93:0/2865314824 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 msgr2=0x7fd4540f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:27.051 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/2865314824 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd4540f6ca0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd444004660 tx=0x7fd444009b30).stop 2021-11-20T09:00:27.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- 172.21.15.93:0/2865314824 shutdown_connections 2021-11-20T09:00:27.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/2865314824 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd4540f6ca0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:27.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/2865314824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4540f55a0 0x7fd4540f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:27.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- 172.21.15.93:0/2865314824 >> 172.21.15.93:0/2865314824 conn(0x7fd4540f0ce0 msgr2=0x7fd4540f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:27.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- 172.21.15.93:0/2865314824 shutdown_connections 2021-11-20T09:00:27.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- 172.21.15.93:0/2865314824 wait complete. 2021-11-20T09:00:27.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 Processor -- start 2021-11-20T09:00:27.053 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- start start 2021-11-20T09:00:27.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4540f55a0 0x7fd454105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:27.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd454106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:27.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd454106840 con 0x7fd4540f6840 2021-11-20T09:00:27.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.047+0000 7fd45b5e6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd454106980 con 0x7fd4540f55a0 2021-11-20T09:00:27.054 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd458b81700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd454106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:27.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd458b81700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd454106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44290/0 (socket says 172.21.15.93:44290) 2021-11-20T09:00:27.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd458b81700 1 -- 172.21.15.93:0/1003017418 learned_addr learned my addr 172.21.15.93:0/1003017418 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:27.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd459382700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4540f55a0 0x7fd454105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:27.055 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd458b81700 1 -- 172.21.15.93:0/1003017418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4540f55a0 msgr2=0x7fd454105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:27.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd458b81700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4540f55a0 0x7fd454105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:27.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd458b81700 1 -- 172.21.15.93:0/1003017418 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd444005040 con 0x7fd4540f6840 2021-11-20T09:00:27.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd458b81700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd454106290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fd444009f50 tx=0x7fd444009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:27.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd449ffb700 1 -- 172.21.15.93:0/1003017418 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd444018070 con 0x7fd4540f6840 2021-11-20T09:00:27.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd449ffb700 1 -- 172.21.15.93:0/1003017418 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd444013410 con 0x7fd4540f6840 2021-11-20T09:00:27.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45405f4b0 con 0x7fd4540f6840 2021-11-20T09:00:27.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd449ffb700 1 -- 172.21.15.93:0/1003017418 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44401c3c0 con 0x7fd4540f6840 2021-11-20T09:00:27.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.048+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45405fa70 con 0x7fd4540f6840 2021-11-20T09:00:27.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.049+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4540fdd30 con 0x7fd4540f6840 2021-11-20T09:00:27.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.049+0000 7fd449ffb700 1 -- 172.21.15.93:0/1003017418 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd44401d420 con 0x7fd4540f6840 2021-11-20T09:00:27.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.050+0000 7fd449ffb700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd44005c2a0 0x7fd44005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:27.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.050+0000 7fd449ffb700 1 -- 172.21.15.93:0/1003017418 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd444077f40 con 0x7fd4540f6840 2021-11-20T09:00:27.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.050+0000 7fd459382700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd44005c2a0 0x7fd44005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:27.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.050+0000 7fd459382700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd44005c2a0 0x7fd44005e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fd4540f66a0 tx=0x7fd45000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:27.059 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.054+0000 7fd449ffb700 1 -- 172.21.15.93:0/1003017418 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4440a42b0 con 0x7fd4540f6840 2021-11-20T09:00:27.240 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.241+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd45410d570 con 0x7fd4540f6840 2021-11-20T09:00:27.241 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.241+0000 7fd449ffb700 1 -- 172.21.15.93:0/1003017418 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7fd44404a800 con 0x7fd4540f6840 2021-11-20T09:00:27.241 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:27.242 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:27.243 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd44005c2a0 msgr2=0x7fd44005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:27.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd44005c2a0 0x7fd44005e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fd4540f66a0 tx=0x7fd45000b040).stop 2021-11-20T09:00:27.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 msgr2=0x7fd454106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:27.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd454106290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fd444009f50 tx=0x7fd444009b30).stop 2021-11-20T09:00:27.244 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 shutdown_connections 2021-11-20T09:00:27.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd44005c2a0 0x7fd44005e760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:27.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4540f55a0 0x7fd454105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:27.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 --2- 172.21.15.93:0/1003017418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4540f6840 0x7fd454106290 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:27.245 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 >> 172.21.15.93:0/1003017418 conn(0x7fd4540f0ce0 msgr2=0x7fd4540fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:27.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 shutdown_connections 2021-11-20T09:00:27.246 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:27.243+0000 7fd45b5e6700 1 -- 172.21.15.93:0/1003017418 wait complete. 2021-11-20T09:00:27.246 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:27.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:27.254 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:27.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:27 smithi102 conmon[36472]: audit 2021-11-20T09:00:27.242470+0000 mon.smithi093 (mon.0) 790 2021-11-20T09:00:27.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:27 smithi102 conmon[36472]: : audit [DBG] from='client.? 172.21.15.93:0/1003017418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:27.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:27 smithi093 conmon[33522]: audit 2021-11-20T09:00:27.242470+0000 mon.smithi093 ( 2021-11-20T09:00:27.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:27 smithi093 conmon[33522]: mon.0) 790 : audit [DBG] from='client.? 172.21.15.93:0/1003017418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:28.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:28.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 -- 172.21.15.93:0/273020151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 msgr2=0x7f62740f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:28.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 --2- 172.21.15.93:0/273020151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f62740f66f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f6268004660 tx=0x7f6268009b30).stop 2021-11-20T09:00:28.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 -- 172.21.15.93:0/273020151 shutdown_connections 2021-11-20T09:00:28.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 --2- 172.21.15.93:0/273020151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f62740f66f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:28.383 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 --2- 172.21.15.93:0/273020151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62740f7c80 0x7f62740f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:28.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 -- 172.21.15.93:0/273020151 >> 172.21.15.93:0/273020151 conn(0x7f62740f1d10 msgr2=0x7f62740f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:28.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 -- 172.21.15.93:0/273020151 shutdown_connections 2021-11-20T09:00:28.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.379+0000 7f627359e700 1 -- 172.21.15.93:0/273020151 wait complete. 2021-11-20T09:00:28.384 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627359e700 1 Processor -- start 2021-11-20T09:00:28.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627359e700 1 -- start start 2021-11-20T09:00:28.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627359e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f627406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:28.385 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627359e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62740f7c80 0x7f6274069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:28.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627359e700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627406a380 con 0x7f62740f6290 2021-11-20T09:00:28.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627359e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627406a4c0 con 0x7f62740f7c80 2021-11-20T09:00:28.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f627406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:28.386 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627259c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f627406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44298/0 (socket says 172.21.15.93:44298) 2021-11-20T09:00:28.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627259c700 1 -- 172.21.15.93:0/1397756261 learned_addr learned my addr 172.21.15.93:0/1397756261 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:28.387 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f6271d9b700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62740f7c80 0x7f6274069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:28.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627259c700 1 -- 172.21.15.93:0/1397756261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62740f7c80 msgr2=0x7f6274069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:28.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627259c700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62740f7c80 0x7f6274069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:28.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627259c700 1 -- 172.21.15.93:0/1397756261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268005040 con 0x7f62740f6290 2021-11-20T09:00:28.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f627259c700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f627406b9a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f6264002700 tx=0x7f6264009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:28.388 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.380+0000 7f6262ffd700 1 -- 172.21.15.93:0/1397756261 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6264009320 con 0x7f62740f6290 2021-11-20T09:00:28.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.381+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627406a7a0 con 0x7f62740f6290 2021-11-20T09:00:28.389 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.381+0000 7f6262ffd700 1 -- 172.21.15.93:0/1397756261 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f626400f040 con 0x7f62740f6290 2021-11-20T09:00:28.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.381+0000 7f6262ffd700 1 -- 172.21.15.93:0/1397756261 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6264014900 con 0x7f62740f6290 2021-11-20T09:00:28.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.381+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627410ee30 con 0x7f62740f6290 2021-11-20T09:00:28.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.381+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6274045ad0 con 0x7f62740f6290 2021-11-20T09:00:28.390 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.382+0000 7f6262ffd700 1 -- 172.21.15.93:0/1397756261 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6264009480 con 0x7f62740f6290 2021-11-20T09:00:28.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.382+0000 7f6262ffd700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f625c05c300 0x7f625c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:28.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.382+0000 7f6262ffd700 1 -- 172.21.15.93:0/1397756261 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f6264076610 con 0x7f62740f6290 2021-11-20T09:00:28.391 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.382+0000 7f6271d9b700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f625c05c300 0x7f625c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:28.392 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.382+0000 7f6271d9b700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f625c05c300 0x7f625c05e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f6268004e30 tx=0x7f6268009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:28.392 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.386+0000 7f6262ffd700 1 -- 172.21.15.93:0/1397756261 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6264048ed0 con 0x7f62740f6290 2021-11-20T09:00:28.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.571+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6274027a00 con 0x7f62740f6290 2021-11-20T09:00:28.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.571+0000 7f6262ffd700 1 -- 172.21.15.93:0/1397756261 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f626404c4f0 con 0x7f62740f6290 2021-11-20T09:00:28.571 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:28.571 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":17}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:14.604253+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24265":{"gid":24265,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":54,"addr":"172.21.15.102:6827/1123372428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1123372428},{"type":"v1","addr":"172.21.15.102:6827","nonce":1123372428}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:28.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f625c05c300 msgr2=0x7f625c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:28.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f625c05c300 0x7f625c05e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f6268004e30 tx=0x7f6268009b30).stop 2021-11-20T09:00:28.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 msgr2=0x7f627406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:28.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f627406b9a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f6264002700 tx=0x7f6264009d80).stop 2021-11-20T09:00:28.574 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 shutdown_connections 2021-11-20T09:00:28.574 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f625c05c300 0x7f625c05e7c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:28.574 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62740f6290 0x7f627406b9a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:28.574 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 --2- 172.21.15.93:0/1397756261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62740f7c80 0x7f6274069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:28.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 >> 172.21.15.93:0/1397756261 conn(0x7f62740f1d10 msgr2=0x7f62740fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:28.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 shutdown_connections 2021-11-20T09:00:28.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:28.573+0000 7f627359e700 1 -- 172.21.15.93:0/1397756261 wait complete. 2021-11-20T09:00:28.576 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 73 2021-11-20T09:00:28.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:14.604253+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24265': {'gid': 24265, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 70, 'state': 'up:clientreplay', 'state_seq': 54, 'addr': '172.21.15.102:6827/1123372428', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 1123372428}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 1123372428}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:28.585 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24265:up:clientreplay 2021-11-20T09:00:28.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:28 smithi102 conmon[36472]: cluster 2021-11-20T09:00:27.575705+0000 mgr.smithi093.qdikkm (mgr.14162) 297 : cluster [DBG] pgmap v272: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:28.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:28 smithi102 conmon[36472]: audit 2021-11-20T09 2021-11-20T09:00:28.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:28 smithi102 conmon[36472]: :00:28.572389+0000 mon.smithi093 (mon.0) 791 : audit [DBG] from='client.? 172.21.15.93:0/1397756261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:28.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:28 smithi093 conmon[33522]: cluster 2021-11-20T09:00:27.575705+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T09:00:28.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:28 smithi093 conmon[33522]: 297 : cluster [DBG] pgmap v272: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:00:28.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:28 smithi093 conmon[33522]: audit 2021-11-20T09:00:28.572389+0000 mon.smithi093 (mon.0) 791 : audit [DBG] from='client.? 172.21.15.93:0/1397756261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:29.586 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:29.724 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 -- 172.21.15.93:0/2727118776 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f6760 msgr2=0x7f73600f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:29.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 --2- 172.21.15.93:0/2727118776 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f6760 0x7f73600f6bc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f7354004660 tx=0x7f7354009b30).stop 2021-11-20T09:00:29.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 -- 172.21.15.93:0/2727118776 shutdown_connections 2021-11-20T09:00:29.725 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 --2- 172.21.15.93:0/2727118776 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f6760 0x7f73600f6bc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:29.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 --2- 172.21.15.93:0/2727118776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73600f54c0 0x7f73600f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:29.726 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 -- 172.21.15.93:0/2727118776 >> 172.21.15.93:0/2727118776 conn(0x7f73600f0c00 msgr2=0x7f73600f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:29.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 -- 172.21.15.93:0/2727118776 shutdown_connections 2021-11-20T09:00:29.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.720+0000 7f7365086700 1 -- 172.21.15.93:0/2727118776 wait complete. 2021-11-20T09:00:29.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f7365086700 1 Processor -- start 2021-11-20T09:00:29.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f7365086700 1 -- start start 2021-11-20T09:00:29.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f7365086700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f54c0 0x7f7360109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:29.729 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f7365086700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73600f6760 0x7f736010bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:29.729 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f7365086700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7360060560 con 0x7f73600f54c0 2021-11-20T09:00:29.729 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f7365086700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73600606a0 con 0x7f73600f6760 2021-11-20T09:00:29.730 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f735ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f54c0 0x7f7360109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:29.730 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f735ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f54c0 0x7f7360109a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44308/0 (socket says 172.21.15.93:44308) 2021-11-20T09:00:29.730 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f735ffff700 1 -- 172.21.15.93:0/1820102841 learned_addr learned my addr 172.21.15.93:0/1820102841 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:29.731 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.721+0000 7f735f7fe700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73600f6760 0x7f736010bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:29.731 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f735ffff700 1 -- 172.21.15.93:0/1820102841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73600f6760 msgr2=0x7f736010bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:29.731 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f735ffff700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73600f6760 0x7f736010bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:29.731 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f735ffff700 1 -- 172.21.15.93:0/1820102841 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7354005040 con 0x7f73600f54c0 2021-11-20T09:00:29.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f735ffff700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f54c0 0x7f7360109a60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f7350002700 tx=0x7f7350009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:29.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f735cff9700 1 -- 172.21.15.93:0/1820102841 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73500079c0 con 0x7f73600f54c0 2021-11-20T09:00:29.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f735cff9700 1 -- 172.21.15.93:0/1820102841 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f735000f040 con 0x7f73600f54c0 2021-11-20T09:00:29.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736010c510 con 0x7f73600f54c0 2021-11-20T09:00:29.733 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f735cff9700 1 -- 172.21.15.93:0/1820102841 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7350014860 con 0x7f73600f54c0 2021-11-20T09:00:29.733 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.722+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736010caa0 con 0x7f73600f54c0 2021-11-20T09:00:29.733 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.723+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73600f48b0 con 0x7f73600f54c0 2021-11-20T09:00:29.733 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.723+0000 7f735cff9700 1 -- 172.21.15.93:0/1820102841 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f735001b060 con 0x7f73600f54c0 2021-11-20T09:00:29.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.724+0000 7f735cff9700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f734805c300 0x7f734805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:29.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.724+0000 7f735cff9700 1 -- 172.21.15.93:0/1820102841 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7350076160 con 0x7f73600f54c0 2021-11-20T09:00:29.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.724+0000 7f735f7fe700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f734805c300 0x7f734805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:29.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.724+0000 7f735f7fe700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f734805c300 0x7f734805e7c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7354009f50 tx=0x7f7354009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:29.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.727+0000 7f735cff9700 1 -- 172.21.15.93:0/1820102841 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73500489c0 con 0x7f73600f54c0 2021-11-20T09:00:29.911 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.912+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7360045ad0 con 0x7f73600f54c0 2021-11-20T09:00:29.912 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.913+0000 7f735cff9700 1 -- 172.21.15.93:0/1820102841 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+2183 (secure 0 0 0) 0x7f735004bfe0 con 0x7f73600f54c0 2021-11-20T09:00:29.913 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:29.913 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:29.623722+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:replay","state_seq":3,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:29.915 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f734805c300 msgr2=0x7f734805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:29.915 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f734805c300 0x7f734805e7c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7354009f50 tx=0x7f7354009b30).stop 2021-11-20T09:00:29.917 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f54c0 msgr2=0x7f7360109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:29.917 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f54c0 0x7f7360109a60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f7350002700 tx=0x7f7350009d80).stop 2021-11-20T09:00:29.917 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 shutdown_connections 2021-11-20T09:00:29.918 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f734805c300 0x7f734805e7c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:29.918 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73600f54c0 0x7f7360109a60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:29.918 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 --2- 172.21.15.93:0/1820102841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73600f6760 0x7f736010bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:29.919 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 >> 172.21.15.93:0/1820102841 conn(0x7f73600f0c00 msgr2=0x7f73600fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:29.919 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 shutdown_connections 2021-11-20T09:00:29.919 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:29.914+0000 7f7365086700 1 -- 172.21.15.93:0/1820102841 wait complete. 2021-11-20T09:00:29.920 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 75 2021-11-20T09:00:29.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:29 smithi102 conmon[36472]: cluster 2021-11-20T09:00:29.415376+0000 mon.smithi093 (mon.0) 792 : cluster [INF] daemon mds.cephfs.smithi102.hjodmi restarted 2021-11-20T09:00:29.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:29 smithi102 conmon[36472]: cluster 2021-11-20T09:00:29.422355+0000 mon.smithi093 (mon.0) 793 : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-20T09:00:29.923 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:29 smithi093 conmon[33522]: debug 2021-11-20T09:00:29.614+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:00:29.924 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:29 smithi093 conmon[33522]: cluster 2021-11-20T09 2021-11-20T09:00:29.924 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:29 smithi093 conmon[33522]: :00:29.415376+0000 mon.smithi093 (mon.0) 792 : cluster [INF] daemon mds.cephfs.smithi102.hjodmi restarted 2021-11-20T09:00:29.926 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:29 smithi093 conmon[33522]: cluster 2021-11-20T09: 2021-11-20T09:00:29.926 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:29 smithi093 conmon[33522]: 00:29.422355+0000 mon.smithi093 (mon.0) 793 : cluster [DBG] 2021-11-20T09:00:29.928 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:29 smithi093 conmon[33522]: osdmap e56: 6 total, 6 up, 6 in 2021-11-20T09:00:29.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:29.623722+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:29.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:replay 2021-11-20T09:00:30.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: cluster 2021-11-20T09:00:29.576029+0000 mgr.smithi093.qdikkm (mgr.14162) 298 : cluster [DBG] pgmap v274: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:00:30.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 2021-11-20T09:00:30.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: cluster 2021-11-20T09:00:29.615133+0000 mon.smithi093 (mon.0) 794 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:00:30.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: cluster 2021-11-20T09:00:29.623574+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:30.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 795 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:boot 2021-11-20T09:00:30.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: cluster 2021-11-20T09 2021-11-20T09:00:30.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: :00:29.623669+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:30.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 796 : cluster [INF] Standby daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 2021-11-20T09:00:30.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: cluster 2021- 2021-11-20T09:00:30.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 11-20T09:00:29.623801+0000 2021-11-20T09:00:30.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: mon.smithi093 (mon.0) 797 : cluster 2021-11-20T09:00:30.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:00:30.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: cluster 2021- 2021-11-20T09:00:30.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 11-20T09:00:29.624056+0000 2021-11-20T09:00:30.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: mon.smithi093 (mon.0) 798 : cluster 2021-11-20T09:00:30.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:00:30.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: audit 2021- 2021-11-20T09:00:30.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 11-20T09:00:29.624215+0000 2021-11-20T09:00:30.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: mon.smithi093 (mon.0) 799 : audit 2021-11-20T09:00:30.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T09:00:30.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: cluster 2021- 2021-11-20T09:00:30.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 11-20T09:00:29.627198+0000 2021-11-20T09:00:30.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: mon.smithi093 (mon.0) 800 : cluster 2021-11-20T09:00:30.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:replay} 1 up:standby 2021-11-20T09:00:30.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: audit 2021- 2021-11-20T09:00:30.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: 11-20T09:00:29.913596 2021-11-20T09:00:30.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 801 2021-11-20T09:00:30.927 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:30 smithi102 conmon[36472]: : audit [DBG] from='client.? 172.21.15.93:0/1820102841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:30.931 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:30.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: cluster 2021-11-20T09:00:29.576029+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T09:00:30.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: 298 : cluster [DBG] pgmap v274: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:00:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: cluster 2021-11-20T09:00:29.615133+0000 mon.smithi093 (mon.0) 794 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:00:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: cluster 2021 2021-11-20T09:00:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: -11-20T09:00:29.623574+0000 mon.smithi093 (mon.0) 795 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:boot 2021-11-20T09:00:30.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T09:00:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: T09:00:29.623669+0000 mon.smithi093 (mon.0) 796 : cluster [INF] Standby daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 2021-11-20T09:00:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: cluster 2021-11-20T09:00:29.623801+0000 mon.smithi093 2021-11-20T09:00:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: (mon.0) 797 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:00:30.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: cluster 2021-11-20T09:00:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: 2021-11-20T09:00:29.624056+0000 mon.smithi093 (mon.0) 798 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:00:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: audit 2021-11-20T09:00:29 2021-11-20T09:00:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: .624215+0000 mon.smithi093 (mon.0) 799 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T09:00:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: cluster 2021-11-20T09:00:29.627198+0000 mon.smithi093 (mon.0) 800 2021-11-20T09:00:30.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:replay} 1 up:standby 2021-11-20T09:00:30.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: audit 2021 2021-11-20T09:00:30.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:30 smithi093 conmon[33522]: -11-20T09:00:29.913596+0000 mon.smithi093 (mon.0) 801 : audit [DBG] from='client.? 172.21.15.93:0/1820102841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:31.068 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 -- 172.21.15.93:0/2820454248 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f55a0 msgr2=0x7f50900f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:31.068 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 --2- 172.21.15.93:0/2820454248 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f55a0 0x7f50900f59c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5080004660 tx=0x7f5080009b30).stop 2021-11-20T09:00:31.069 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 -- 172.21.15.93:0/2820454248 shutdown_connections 2021-11-20T09:00:31.069 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 --2- 172.21.15.93:0/2820454248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50900f6840 0x7f50900f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:31.069 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 --2- 172.21.15.93:0/2820454248 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f55a0 0x7f50900f59c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:31.069 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 -- 172.21.15.93:0/2820454248 >> 172.21.15.93:0/2820454248 conn(0x7f50900f0ce0 msgr2=0x7f50900f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:31.070 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 -- 172.21.15.93:0/2820454248 shutdown_connections 2021-11-20T09:00:31.070 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.064+0000 7f5095422700 1 -- 172.21.15.93:0/2820454248 wait complete. 2021-11-20T09:00:31.070 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f5095422700 1 Processor -- start 2021-11-20T09:00:31.070 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f5095422700 1 -- start start 2021-11-20T09:00:31.071 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f5095422700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50900f55a0 0x7f509005ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:31.071 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f5095422700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f6840 0x7f509010bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:31.071 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f5095422700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509010c0e0 con 0x7f50900f6840 2021-11-20T09:00:31.071 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f5095422700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509010c220 con 0x7f50900f55a0 2021-11-20T09:00:31.072 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508e7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f6840 0x7f509010bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:31.072 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508e7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f6840 0x7f509010bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44320/0 (socket says 172.21.15.93:44320) 2021-11-20T09:00:31.072 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508e7fc700 1 -- 172.21.15.93:0/3761419777 learned_addr learned my addr 172.21.15.93:0/3761419777 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:31.072 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508effd700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50900f55a0 0x7f509005ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:31.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508e7fc700 1 -- 172.21.15.93:0/3761419777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50900f55a0 msgr2=0x7f509005ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:31.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508e7fc700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50900f55a0 0x7f509005ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:31.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508e7fc700 1 -- 172.21.15.93:0/3761419777 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5080005040 con 0x7f50900f6840 2021-11-20T09:00:31.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f508e7fc700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f6840 0x7f509010bbf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f508400e7f0 tx=0x7f508400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:31.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.065+0000 7f50777fe700 1 -- 172.21.15.93:0/3761419777 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50840092e0 con 0x7f50900f6840 2021-11-20T09:00:31.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.066+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509010c4b0 con 0x7f50900f6840 2021-11-20T09:00:31.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.066+0000 7f50777fe700 1 -- 172.21.15.93:0/3761419777 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f508400f040 con 0x7f50900f6840 2021-11-20T09:00:31.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.066+0000 7f50777fe700 1 -- 172.21.15.93:0/3761419777 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50840146c0 con 0x7f50900f6840 2021-11-20T09:00:31.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.066+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509010ca70 con 0x7f50900f6840 2021-11-20T09:00:31.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.067+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5090027a00 con 0x7f50900f6840 2021-11-20T09:00:31.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.067+0000 7f50777fe700 1 -- 172.21.15.93:0/3761419777 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f508401b030 con 0x7f50900f6840 2021-11-20T09:00:31.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.067+0000 7f50777fe700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f507805c2a0 0x7f507805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:31.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.067+0000 7f50777fe700 1 -- 172.21.15.93:0/3761419777 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f50840760e0 con 0x7f50900f6840 2021-11-20T09:00:31.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.067+0000 7f508effd700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f507805c2a0 0x7f507805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:31.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.068+0000 7f508effd700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f507805c2a0 0x7f507805e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f50800043d0 tx=0x7f508000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:31.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.071+0000 7f50777fe700 1 -- 172.21.15.93:0/3761419777 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5084048990 con 0x7f50900f6840 2021-11-20T09:00:31.254 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.255+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50900fdd30 con 0x7f50900f6840 2021-11-20T09:00:31.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.256+0000 7f50777fe700 1 -- 172.21.15.93:0/3761419777 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f5084019090 con 0x7f50900f6840 2021-11-20T09:00:31.255 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:31.256 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:30.625020+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:reconnect","state_seq":50,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:31.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f507805c2a0 msgr2=0x7f507805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:31.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f507805c2a0 0x7f507805e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f50800043d0 tx=0x7f508000b040).stop 2021-11-20T09:00:31.259 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f6840 msgr2=0x7f509010bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:31.259 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f6840 0x7f509010bbf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f508400e7f0 tx=0x7f508400b040).stop 2021-11-20T09:00:31.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 shutdown_connections 2021-11-20T09:00:31.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f507805c2a0 0x7f507805e760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:31.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50900f55a0 0x7f509005ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:31.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 --2- 172.21.15.93:0/3761419777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50900f6840 0x7f509010bbf0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:31.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.257+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 >> 172.21.15.93:0/3761419777 conn(0x7f50900f0ce0 msgr2=0x7f50900fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:31.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.258+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 shutdown_connections 2021-11-20T09:00:31.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:31.258+0000 7f5095422700 1 -- 172.21.15.93:0/3761419777 wait complete. 2021-11-20T09:00:31.263 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 76 2021-11-20T09:00:31.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:30.625020+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:reconnect', 'state_seq': 50, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:31.270 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:reconnect 2021-11-20T09:00:31.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: cluster 2021-11-20T09:00:30.627825+0000 mon.smithi093 (mon.0) 802 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:reconnect 2021-11-20T09:00:31.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: cluster 2021-11-20T09:00:30.627924+0000 2021-11-20T09:00:31.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: mon.smithi093 (mon.0) 803 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:reconnect} 1 up:standby 2021-11-20T09:00:31.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: cluster 2021-11-20T09:00:30.648099+0000 mds.cephfs.smithi102.lhrlmh (mds.0) 1 : 2021-11-20T09:00:31.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:00:31.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: audit 2021-11-20T09:00:31.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: 2021-11-20T09:00 2021-11-20T09:00:31.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: :31.256937+0000 mon.smithi093 (mon. 2021-11-20T09:00:31.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:31 smithi102 conmon[36472]: 0) 804 : audit [DBG] from='client.? 172.21.15.93:0/3761419777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:31.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: cluster 2021-11-20T09:00:30.627825 2021-11-20T09:00:31.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 802 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:reconnect 2021-11-20T09:00:31.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: cluster 2021-11-20T09:00: 2021-11-20T09:00:31.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: 30.627924+0000 mon.smithi093 (mon.0) 803 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:reconnect} 1 up:standby 2021-11-20T09:00:31.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T09:00:31.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: 20T09:00:30.648099+0000 mds.cephfs.smithi102.lhrlmh (mds 2021-11-20T09:00:31.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: .0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:00:31.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: audit 2021-11- 2021-11-20T09:00:31.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: 20T09:00:31.256937+0000 mon.smithi093 (mon.0) 804 : audit [DBG] 2021-11-20T09:00:31.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:31 smithi093 conmon[33522]: from='client.? 172.21.15.93:0/3761419777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:32.273 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:32.399 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 -- 172.21.15.93:0/2455581247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 msgr2=0x7f040c0f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:32.400 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 --2- 172.21.15.93:0/2455581247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c0f7da0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f0400004660 tx=0x7f0400009b30).stop 2021-11-20T09:00:32.401 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 -- 172.21.15.93:0/2455581247 shutdown_connections 2021-11-20T09:00:32.401 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 --2- 172.21.15.93:0/2455581247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c0f7da0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:32.402 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 --2- 172.21.15.93:0/2455581247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f040c0f66a0 0x7f040c0f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:32.402 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 -- 172.21.15.93:0/2455581247 >> 172.21.15.93:0/2455581247 conn(0x7f040c0f1da0 msgr2=0x7f040c0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:32.402 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 -- 172.21.15.93:0/2455581247 shutdown_connections 2021-11-20T09:00:32.403 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.395+0000 7f0413e32700 1 -- 172.21.15.93:0/2455581247 wait complete. 2021-11-20T09:00:32.403 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0413e32700 1 Processor -- start 2021-11-20T09:00:32.403 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0413e32700 1 -- start start 2021-11-20T09:00:32.404 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0413e32700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f040c0f66a0 0x7f040c106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:32.404 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0413e32700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:32.404 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0413e32700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c107960 con 0x7f040c0f7940 2021-11-20T09:00:32.404 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0413e32700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c107aa0 con 0x7f040c0f66a0 2021-11-20T09:00:32.405 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f04113cd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:32.405 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f04113cd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c1073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44330/0 (socket says 172.21.15.93:44330) 2021-11-20T09:00:32.406 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f04113cd700 1 -- 172.21.15.93:0/2289940343 learned_addr learned my addr 172.21.15.93:0/2289940343 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:32.406 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0411bce700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f040c0f66a0 0x7f040c106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:32.406 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f04113cd700 1 -- 172.21.15.93:0/2289940343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f040c0f66a0 msgr2=0x7f040c106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:32.406 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f04113cd700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f040c0f66a0 0x7f040c106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:32.407 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f04113cd700 1 -- 172.21.15.93:0/2289940343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400005040 con 0x7f040c0f7940 2021-11-20T09:00:32.407 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f0411bce700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f040c0f66a0 0x7f040c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:00:32.407 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.396+0000 7f04113cd700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c1073b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0400009830 tx=0x7f0400009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:32.408 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.397+0000 7f040a7fc700 1 -- 172.21.15.93:0/2289940343 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0400018070 con 0x7f040c0f7940 2021-11-20T09:00:32.408 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.397+0000 7f040a7fc700 1 -- 172.21.15.93:0/2289940343 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0400013410 con 0x7f040c0f7940 2021-11-20T09:00:32.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.397+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c10c4e0 con 0x7f040c0f7940 2021-11-20T09:00:32.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.397+0000 7f040a7fc700 1 -- 172.21.15.93:0/2289940343 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f040001c620 con 0x7f040c0f7940 2021-11-20T09:00:32.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.397+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c10caa0 con 0x7f040c0f7940 2021-11-20T09:00:32.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.397+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040c027a00 con 0x7f040c0f7940 2021-11-20T09:00:32.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.398+0000 7f040a7fc700 1 -- 172.21.15.93:0/2289940343 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f040001c8e0 con 0x7f040c0f7940 2021-11-20T09:00:32.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.398+0000 7f040a7fc700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f03f405c2a0 0x7f03f405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:32.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.398+0000 7f040a7fc700 1 -- 172.21.15.93:0/2289940343 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0400078200 con 0x7f040c0f7940 2021-11-20T09:00:32.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.398+0000 7f0411bce700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f03f405c2a0 0x7f03f405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:32.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.399+0000 7f0411bce700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f03f405c2a0 0x7f03f405e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f040c0f77a0 tx=0x7f03fc009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:32.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.402+0000 7f040a7fc700 1 -- 172.21.15.93:0/2289940343 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f040004aa00 con 0x7f040c0f7940 2021-11-20T09:00:32.586 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.587+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f040c10c670 con 0x7f040c0f7940 2021-11-20T09:00:32.586 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.587+0000 7f040a7fc700 1 -- 172.21.15.93:0/2289940343 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2184 (secure 0 0 0) 0x7f040c10c670 con 0x7f040c0f7940 2021-11-20T09:00:32.587 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:32.587 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:31.626447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:rejoin","state_seq":51,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:32.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f03f405c2a0 msgr2=0x7f03f405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:32.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f03f405c2a0 0x7f03f405e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f040c0f77a0 tx=0x7f03fc009040).stop 2021-11-20T09:00:32.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 msgr2=0x7f040c1073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:32.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c1073b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0400009830 tx=0x7f0400009b30).stop 2021-11-20T09:00:32.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 shutdown_connections 2021-11-20T09:00:32.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f03f405c2a0 0x7f03f405e760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:32.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f040c0f66a0 0x7f040c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:32.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 --2- 172.21.15.93:0/2289940343 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040c0f7940 0x7f040c1073b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:32.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 >> 172.21.15.93:0/2289940343 conn(0x7f040c0f1da0 msgr2=0x7f040c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:32.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 shutdown_connections 2021-11-20T09:00:32.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:32.589+0000 7f0413e32700 1 -- 172.21.15.93:0/2289940343 wait complete. 2021-11-20T09:00:32.591 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 77 2021-11-20T09:00:32.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:31.626447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:rejoin', 'state_seq': 51, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:32.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:rejoin 2021-11-20T09:00:32.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:32 smithi102 conmon[36472]: cluster 2021-11-20T09:00:31.576434+0000 mgr.smithi093.qdikkm (mgr.14162) 299 2021-11-20T09:00:32.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:32 smithi102 conmon[36472]: : cluster [DBG] pgmap v275: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T09:00:32.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:32 smithi102 conmon[36472]: cluster 2021-11-20T09:00:31.632961+0000 mon.smithi093 (mon.0) 805 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:rejoin 2021-11-20T09:00:32.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:32 smithi102 conmon[36472]: cluster 2021-11-20T09:00:31.633024+0000 mon.smithi093 (mon.0) 806 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:rejoin} 1 up:standby 2021-11-20T09:00:32.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:32 smithi102 conmon[36472]: audit 2021-11-20T09:00:32.588190+0000 mon.smithi093 (mon.0) 807 : audit [DBG] from='client.? 172.21.15.93:0/2289940343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:32.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: cluster 2021-11-20T09:00:31.576434+0000 mgr.smithi093.qdikkm (mgr.14162) 299 : cluster 2021-11-20T09:00:32.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: [DBG] pgmap v275: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T09:00:32.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: cluster 2021-11-20T09:00:31.632961+0000 mon.smithi093 (mon.0) 805 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:rejoin 2021-11-20T09:00:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: cluster 2021-11-20T09:00:31.633024+0000 mon.smithi093 ( 2021-11-20T09:00:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: mon.0) 806 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:rejoin} 1 up:standby 2021-11-20T09:00:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: 2021-11-20T09:00:32.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: audit 2021-11-20T09:00: 2021-11-20T09:00:32.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: 32.588190+0000 2021-11-20T09:00:32.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: mon.smithi093 (mon.0) 807 : audit [DBG] 2021-11-20T09:00:32.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:32 smithi093 conmon[33522]: from='client.? 172.21.15.93:0/2289940343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:33.602 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:33.727 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- 172.21.15.93:0/4257458005 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f5580 msgr2=0x7f90300f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:33.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 --2- 172.21.15.93:0/4257458005 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f5580 0x7f90300f59a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f902c004660 tx=0x7f902c009b30).stop 2021-11-20T09:00:33.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- 172.21.15.93:0/4257458005 shutdown_connections 2021-11-20T09:00:33.728 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 --2- 172.21.15.93:0/4257458005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90300f6820 0x7f90300f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 --2- 172.21.15.93:0/4257458005 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f5580 0x7f90300f59a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:33.730 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- 172.21.15.93:0/4257458005 >> 172.21.15.93:0/4257458005 conn(0x7f90300f0ce0 msgr2=0x7f90300f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:33.731 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- 172.21.15.93:0/4257458005 shutdown_connections 2021-11-20T09:00:33.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- 172.21.15.93:0/4257458005 wait complete. 2021-11-20T09:00:33.732 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 Processor -- start 2021-11-20T09:00:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- start start 2021-11-20T09:00:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90300f5580 0x7f90300f9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f6820 0x7f90300f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:33.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90300f9ab0 con 0x7f90300f6820 2021-11-20T09:00:33.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9037609700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90300f8050 con 0x7f90300f5580 2021-11-20T09:00:33.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9034ba4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f6820 0x7f90300f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:33.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.724+0000 7f9034ba4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f6820 0x7f90300f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44342/0 (socket says 172.21.15.93:44342) 2021-11-20T09:00:33.734 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9034ba4700 1 -- 172.21.15.93:0/3291200201 learned_addr learned my addr 172.21.15.93:0/3291200201 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:33.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f90353a5700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90300f5580 0x7f90300f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:33.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9034ba4700 1 -- 172.21.15.93:0/3291200201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90300f5580 msgr2=0x7f90300f9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:33.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9034ba4700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90300f5580 0x7f90300f9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:33.735 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9034ba4700 1 -- 172.21.15.93:0/3291200201 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c005040 con 0x7f90300f6820 2021-11-20T09:00:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9034ba4700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f6820 0x7f90300f7b60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f902000c790 tx=0x7f9020007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9025ffb700 1 -- 172.21.15.93:0/3291200201 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f902000e680 con 0x7f90300f6820 2021-11-20T09:00:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90300f82d0 con 0x7f90300f6820 2021-11-20T09:00:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9025ffb700 1 -- 172.21.15.93:0/3291200201 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f902000ec90 con 0x7f90300f6820 2021-11-20T09:00:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9025ffb700 1 -- 172.21.15.93:0/3291200201 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9020012cf0 con 0x7f90300f6820 2021-11-20T09:00:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.725+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90300f8890 con 0x7f90300f6820 2021-11-20T09:00:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.726+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9030046330 con 0x7f90300f6820 2021-11-20T09:00:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.726+0000 7f9025ffb700 1 -- 172.21.15.93:0/3291200201 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f902001a060 con 0x7f90300f6820 2021-11-20T09:00:33.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.727+0000 7f9025ffb700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f901c05c2a0 0x7f901c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:33.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.727+0000 7f9025ffb700 1 -- 172.21.15.93:0/3291200201 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f90200761a0 con 0x7f90300f6820 2021-11-20T09:00:33.738 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.727+0000 7f90353a5700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f901c05c2a0 0x7f901c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.727+0000 7f90353a5700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f901c05c2a0 0x7f901c05e760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f902c004e30 tx=0x7f902c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:33.740 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.730+0000 7f9025ffb700 1 -- 172.21.15.93:0/3291200201 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9020048a00 con 0x7f90300f6820 2021-11-20T09:00:33.913 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.914+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90300fdd10 con 0x7f90300f6820 2021-11-20T09:00:33.913 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.915+0000 7f9025ffb700 1 -- 172.21.15.93:0/3291200201 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7f9020018070 con 0x7f90300f6820 2021-11-20T09:00:33.914 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:33.916 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f901c05c2a0 msgr2=0x7f901c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:33.917 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f901c05c2a0 0x7f901c05e760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f902c004e30 tx=0x7f902c009b30).stop 2021-11-20T09:00:33.917 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f6820 msgr2=0x7f90300f7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:33.917 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f6820 0x7f90300f7b60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f902000c790 tx=0x7f9020007d70).stop 2021-11-20T09:00:33.918 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 shutdown_connections 2021-11-20T09:00:33.918 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f901c05c2a0 0x7f901c05e760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:33.919 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90300f5580 0x7f90300f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:33.919 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 --2- 172.21.15.93:0/3291200201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90300f6820 0x7f90300f7b60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:33.919 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 >> 172.21.15.93:0/3291200201 conn(0x7f90300f0ce0 msgr2=0x7f9030060260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:33.920 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 shutdown_connections 2021-11-20T09:00:33.920 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:33.916+0000 7f9037609700 1 -- 172.21.15.93:0/3291200201 wait complete. 2021-11-20T09:00:33.921 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:33.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:33 smithi102 conmon[36472]: cluster 2021-11-20T09:00:32.634695+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:33.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:33 smithi102 conmon[36472]: 808 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:clientreplay 2021-11-20T09:00:33.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:33 smithi102 conmon[36472]: cluster 2021-11-20T09:00:32.634751+0000 mon.smithi093 (mon. 2021-11-20T09:00:33.923 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:33 smithi102 conmon[36472]: 0) 809 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:00:33.927 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:33 smithi093 conmon[33522]: cluster 2021-11-20T09:00:32.634695+0000 mon.smithi093 ( 2021-11-20T09:00:33.927 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:33 smithi093 conmon[33522]: mon.0) 808 : cluster [DBG] mds.? [v2:172.21.15.102:6824/1330867096,v1:172.21.15.102:6825/1330867096] up:clientreplay 2021-11-20T09:00:33.929 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:33 smithi093 conmon[33522]: cluster 2021-11-20T09:00:32.634751+0000 mon.smithi093 (mon. 2021-11-20T09:00:33.929 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:33 smithi093 conmon[33522]: 0) 809 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:00:33.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:33.932 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:34.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: cluster 2021-11-20T09:00:33.577021+0000 mgr.smithi093.qdikkm (mgr.14162) 300 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T09:00:34.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: cluster 2021-11-20T09:00:33.644261+0000 mon.smithi093 (mon.0) 810 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:standby 2021-11-20T09:00:34.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: cluster 2021-11-20T09:00:33.644337+0000 2021-11-20T09:00:34.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: mon.smithi093 (mon.0) 811 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:00:34.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: audit 2021-11-20T09:00:34.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: 2021-11-20T09:00: 2021-11-20T09:00:34.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: 33.915584+0000 mon.smithi093 (mon.0 2021-11-20T09:00:34.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:34 smithi102 conmon[36472]: ) 812 : audit [DBG] from='client.? 172.21.15.93:0/3291200201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:34.938 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:34.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: cluster 2021-11-20T09:00:33.577021+0000 mgr.smithi093.qdikkm (mgr.14162) 300 : 2021-11-20T09:00:34.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T09:00:34.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: cluster 2021-11-20T09:00:33.644261+0000 mon.smithi093 (mon.0) 810 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:standby 2021-11-20T09:00:34.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: cluster 2021-11-20T09 2021-11-20T09:00:34.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: :00:33.644337+0000 mon.smithi093 ( 2021-11-20T09:00:34.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: mon.0) 811 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:00:34.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: 2021-11-20T09:00:34.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: audit 2021-11-20T09:00:34.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: 2021-11-20T09:00:33.915584+0000 mon.smithi093 (mon. 2021-11-20T09:00:34.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:34 smithi093 conmon[33522]: 0) 812 : audit [DBG] from='client.? 172.21.15.93:0/3291200201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:35.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- 172.21.15.93:0/2097402051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f5300 msgr2=0x7fd1980f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:35.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 --2- 172.21.15.93:0/2097402051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f5300 0x7fd1980f7a80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd188004660 tx=0x7fd188009b30).stop 2021-11-20T09:00:35.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- 172.21.15.93:0/2097402051 shutdown_connections 2021-11-20T09:00:35.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 --2- 172.21.15.93:0/2097402051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f5300 0x7fd1980f7a80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:35.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 --2- 172.21.15.93:0/2097402051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1980f4960 0x7fd1980f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:35.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- 172.21.15.93:0/2097402051 >> 172.21.15.93:0/2097402051 conn(0x7fd1980f0440 msgr2=0x7fd1980f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:35.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- 172.21.15.93:0/2097402051 shutdown_connections 2021-11-20T09:00:35.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- 172.21.15.93:0/2097402051 wait complete. 2021-11-20T09:00:35.076 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 Processor -- start 2021-11-20T09:00:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- start start 2021-11-20T09:00:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f4960 0x7fd198106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:35.077 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1980f5300 0x7fd198107370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:35.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd198107920 con 0x7fd1980f4960 2021-11-20T09:00:35.080 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.066+0000 7fd19ff68700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd198107a60 con 0x7fd1980f5300 2021-11-20T09:00:35.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19dd04700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f4960 0x7fd198106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:35.081 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19dd04700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f4960 0x7fd198106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44352/0 (socket says 172.21.15.93:44352) 2021-11-20T09:00:35.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19dd04700 1 -- 172.21.15.93:0/1837356824 learned_addr learned my addr 172.21.15.93:0/1837356824 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:35.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19d503700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1980f5300 0x7fd198107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:35.082 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19dd04700 1 -- 172.21.15.93:0/1837356824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1980f5300 msgr2=0x7fd198107370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:35.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19dd04700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1980f5300 0x7fd198107370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:35.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19dd04700 1 -- 172.21.15.93:0/1837356824 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd188005040 con 0x7fd1980f4960 2021-11-20T09:00:35.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19dd04700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f4960 0x7fd198106e80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fd19400e7c0 tx=0x7fd19400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:35.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd18e7fc700 1 -- 172.21.15.93:0/1837356824 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1940092e0 con 0x7fd1980f4960 2021-11-20T09:00:35.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd18e7fc700 1 -- 172.21.15.93:0/1837356824 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd19400f040 con 0x7fd1980f4960 2021-11-20T09:00:35.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19810c500 con 0x7fd1980f4960 2021-11-20T09:00:35.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd18e7fc700 1 -- 172.21.15.93:0/1837356824 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1940147e0 con 0x7fd1980f4960 2021-11-20T09:00:35.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.067+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19810cac0 con 0x7fd1980f4960 2021-11-20T09:00:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.068+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd198027a00 con 0x7fd1980f4960 2021-11-20T09:00:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.068+0000 7fd18e7fc700 1 -- 172.21.15.93:0/1837356824 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd19401b030 con 0x7fd1980f4960 2021-11-20T09:00:35.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.069+0000 7fd18e7fc700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd18405c2a0 0x7fd18405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:35.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.069+0000 7fd18e7fc700 1 -- 172.21.15.93:0/1837356824 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd1940763f0 con 0x7fd1980f4960 2021-11-20T09:00:35.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.069+0000 7fd19d503700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd18405c2a0 0x7fd18405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:35.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.069+0000 7fd19d503700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd18405c2a0 0x7fd18405e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fd188005010 tx=0x7fd18800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:35.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.073+0000 7fd18e7fc700 1 -- 172.21.15.93:0/1837356824 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd194048c50 con 0x7fd1980f4960 2021-11-20T09:00:35.264 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.261+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1980fee50 con 0x7fd1980f4960 2021-11-20T09:00:35.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.261+0000 7fd18e7fc700 1 -- 172.21.15.93:0/1837356824 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7fd19404c270 con 0x7fd1980f4960 2021-11-20T09:00:35.266 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:35.266 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:35.269 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd18405c2a0 msgr2=0x7fd18405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:35.269 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd18405c2a0 0x7fd18405e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fd188005010 tx=0x7fd18800b040).stop 2021-11-20T09:00:35.270 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f4960 msgr2=0x7fd198106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:35.270 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f4960 0x7fd198106e80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fd19400e7c0 tx=0x7fd19400b040).stop 2021-11-20T09:00:35.271 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 shutdown_connections 2021-11-20T09:00:35.271 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd18405c2a0 0x7fd18405e760 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:35.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1980f4960 0x7fd198106e80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:35.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 --2- 172.21.15.93:0/1837356824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1980f5300 0x7fd198107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:35.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 >> 172.21.15.93:0/1837356824 conn(0x7fd1980f0440 msgr2=0x7fd1980fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:35.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 shutdown_connections 2021-11-20T09:00:35.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:35.263+0000 7fd19ff68700 1 -- 172.21.15.93:0/1837356824 wait complete. 2021-11-20T09:00:35.274 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:35.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:35.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:35.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:35 smithi102 conmon[36472]: audit 2021-11-20T09:00:35.262397+0000 mon.smithi093 (mon.0) 813 : audit [DBG] from='client.? 172.21.15.93:0/1837356824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:35.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:35 smithi093 conmon[33522]: audit 2021-11-20T09:00:35.262397+0000 mon.smithi093 (mon. 2021-11-20T09:00:35.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:35 smithi093 conmon[33522]: 0) 813 : audit [DBG] from='client.? 172.21.15.93:0/1837356824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:36.280 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:36.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 -- 172.21.15.93:0/2939914838 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 msgr2=0x7f81140f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:36.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 --2- 172.21.15.93:0/2939914838 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f81140f6c80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f8108004660 tx=0x7f8108009b30).stop 2021-11-20T09:00:36.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 -- 172.21.15.93:0/2939914838 shutdown_connections 2021-11-20T09:00:36.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 --2- 172.21.15.93:0/2939914838 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f81140f6c80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:36.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 --2- 172.21.15.93:0/2939914838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81140f5580 0x7f81140f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:36.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 -- 172.21.15.93:0/2939914838 >> 172.21.15.93:0/2939914838 conn(0x7f81140f0ce0 msgr2=0x7f81140f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:36.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 -- 172.21.15.93:0/2939914838 shutdown_connections 2021-11-20T09:00:36.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 -- 172.21.15.93:0/2939914838 wait complete. 2021-11-20T09:00:36.426 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 Processor -- start 2021-11-20T09:00:36.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 -- start start 2021-11-20T09:00:36.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81140f5580 0x7f8114060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:36.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f811410bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:36.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.407+0000 7f811b828700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811410c190 con 0x7f81140f6820 2021-11-20T09:00:36.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f811b828700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811410c2d0 con 0x7f81140f5580 2021-11-20T09:00:36.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8118dc3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f811410bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:36.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8118dc3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f811410bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44366/0 (socket says 172.21.15.93:44366) 2021-11-20T09:00:36.429 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8118dc3700 1 -- 172.21.15.93:0/174569756 learned_addr learned my addr 172.21.15.93:0/174569756 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:36.430 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f81195c4700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81140f5580 0x7f8114060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:36.431 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8118dc3700 1 -- 172.21.15.93:0/174569756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81140f5580 msgr2=0x7f8114060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:36.432 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8118dc3700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81140f5580 0x7f8114060550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:36.439 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8118dc3700 1 -- 172.21.15.93:0/174569756 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108005040 con 0x7f81140f6820 2021-11-20T09:00:36.439 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8118dc3700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f811410bca0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f8108009f50 tx=0x7f8108009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:36.439 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8111ffb700 1 -- 172.21.15.93:0/174569756 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8108018070 con 0x7f81140f6820 2021-11-20T09:00:36.439 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8111ffb700 1 -- 172.21.15.93:0/174569756 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8108013410 con 0x7f81140f6820 2021-11-20T09:00:36.439 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811410c4d0 con 0x7f81140f6820 2021-11-20T09:00:36.440 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f8111ffb700 1 -- 172.21.15.93:0/174569756 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f810801c3c0 con 0x7f81140f6820 2021-11-20T09:00:36.440 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.408+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811410ca00 con 0x7f81140f6820 2021-11-20T09:00:36.440 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.409+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8114046330 con 0x7f81140f6820 2021-11-20T09:00:36.440 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.410+0000 7f8111ffb700 1 -- 172.21.15.93:0/174569756 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f810801d420 con 0x7f81140f6820 2021-11-20T09:00:36.441 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.410+0000 7f8111ffb700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f80fc05c2a0 0x7f80fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:36.441 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.410+0000 7f8111ffb700 1 -- 172.21.15.93:0/174569756 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8108078020 con 0x7f81140f6820 2021-11-20T09:00:36.441 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.410+0000 7f81195c4700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f80fc05c2a0 0x7f80fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:36.441 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.410+0000 7f81195c4700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f80fc05c2a0 0x7f80fc05e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f81140f6680 tx=0x7f8104004d00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:36.442 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.414+0000 7f8111ffb700 1 -- 172.21.15.93:0/174569756 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f810804a880 con 0x7f81140f6820 2021-11-20T09:00:36.600 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.602+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8114027a00 con 0x7f81140f6820 2021-11-20T09:00:36.601 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.603+0000 7f8111ffb700 1 -- 172.21.15.93:0/174569756 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7f810804dea0 con 0x7f81140f6820 2021-11-20T09:00:36.602 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:36.603 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:36.604 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f80fc05c2a0 msgr2=0x7f80fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:36.604 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f80fc05c2a0 0x7f80fc05e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f81140f6680 tx=0x7f8104004d00).stop 2021-11-20T09:00:36.604 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 msgr2=0x7f811410bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:36.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f811410bca0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f8108009f50 tx=0x7f8108009b30).stop 2021-11-20T09:00:36.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 shutdown_connections 2021-11-20T09:00:36.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f80fc05c2a0 0x7f80fc05e760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:36.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81140f5580 0x7f8114060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:36.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 --2- 172.21.15.93:0/174569756 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81140f6820 0x7f811410bca0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:36.606 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 >> 172.21.15.93:0/174569756 conn(0x7f81140f0ce0 msgr2=0x7f81140fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:36.606 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 shutdown_connections 2021-11-20T09:00:36.607 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:36.605+0000 7f811b828700 1 -- 172.21.15.93:0/174569756 wait complete. 2021-11-20T09:00:36.607 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:36.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:36.627 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:36.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:36 smithi102 conmon[36472]: cluster 2021-11-20T09:00:35.577602+0000 mgr.smithi093.qdikkm (mgr.14162) 301 : cluster [DBG] pgmap v277: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T09:00:36.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:36 smithi102 conmon[36472]: audit 2021-11-20T09:00:36.604170+0000 mon.smithi093 (mon.0) 814 : audit [DBG] from='client.? 172.21.15.93:0/174569756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:36.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:36 smithi093 conmon[33522]: cluster 2021-11-20T09:00:35.577602+0000 mgr.smithi093.qdikkm ( 2021-11-20T09:00:36.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:36 smithi093 conmon[33522]: mgr.14162) 301 : cluster [DBG] pgmap v277: 65 pgs: 65 active+clean; 3.2 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T09:00:36.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:36 smithi093 conmon[33522]: audit 2021-11-20T09:00:36.604170+0000 mon.smithi093 (mon.0) 814 : audit [DBG] from='client.? 172.21.15.93:0/174569756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:37.629 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:37.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 -- 172.21.15.93:0/40431519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f5950 msgr2=0x7fc6140f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:37.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 --2- 172.21.15.93:0/40431519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f5950 0x7fc6140f5dd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fc600004660 tx=0x7fc600009b30).stop 2021-11-20T09:00:37.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 -- 172.21.15.93:0/40431519 shutdown_connections 2021-11-20T09:00:37.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 --2- 172.21.15.93:0/40431519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f5950 0x7fc6140f5dd0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:37.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 --2- 172.21.15.93:0/40431519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6140f6ca0 0x7fc6140f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:37.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 -- 172.21.15.93:0/40431519 >> 172.21.15.93:0/40431519 conn(0x7fc6140f04a0 msgr2=0x7fc6140f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:37.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 -- 172.21.15.93:0/40431519 shutdown_connections 2021-11-20T09:00:37.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 -- 172.21.15.93:0/40431519 wait complete. 2021-11-20T09:00:37.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 Processor -- start 2021-11-20T09:00:37.783 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.753+0000 7fc61b114700 1 -- start start 2021-11-20T09:00:37.783 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc61b114700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6140f5950 0x7fc6140f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:37.784 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc61b114700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f6ca0 0x7fc61410bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:37.784 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc61b114700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61410c230 con 0x7fc6140f6ca0 2021-11-20T09:00:37.784 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc61b114700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61410c370 con 0x7fc6140f5950 2021-11-20T09:00:37.784 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc613fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f6ca0 0x7fc61410bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:37.784 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc613fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f6ca0 0x7fc61410bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44378/0 (socket says 172.21.15.93:44378) 2021-11-20T09:00:37.786 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc613fff700 1 -- 172.21.15.93:0/2400801732 learned_addr learned my addr 172.21.15.93:0/2400801732 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:37.787 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc618eb0700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6140f5950 0x7fc6140f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:37.788 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc613fff700 1 -- 172.21.15.93:0/2400801732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6140f5950 msgr2=0x7fc6140f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:37.788 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc613fff700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6140f5950 0x7fc6140f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:37.790 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc613fff700 1 -- 172.21.15.93:0/2400801732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc600005040 con 0x7fc6140f6ca0 2021-11-20T09:00:37.791 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc618eb0700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6140f5950 0x7fc6140f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:00:37.791 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc613fff700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f6ca0 0x7fc61410bd40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fc600007a60 tx=0x7fc600009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:37.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc6117fa700 1 -- 172.21.15.93:0/2400801732 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc600018070 con 0x7fc6140f6ca0 2021-11-20T09:00:37.794 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc6117fa700 1 -- 172.21.15.93:0/2400801732 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc600013410 con 0x7fc6140f6ca0 2021-11-20T09:00:37.795 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61410c570 con 0x7fc6140f6ca0 2021-11-20T09:00:37.795 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc6117fa700 1 -- 172.21.15.93:0/2400801732 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc60001c620 con 0x7fc6140f6ca0 2021-11-20T09:00:37.796 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.754+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61410cad0 con 0x7fc6140f6ca0 2021-11-20T09:00:37.796 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.755+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc614027a00 con 0x7fc6140f6ca0 2021-11-20T09:00:37.797 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.756+0000 7fc6117fa700 1 -- 172.21.15.93:0/2400801732 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc60001c890 con 0x7fc6140f6ca0 2021-11-20T09:00:37.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.756+0000 7fc6117fa700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc60405c2a0 0x7fc60405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:37.798 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.756+0000 7fc6117fa700 1 -- 172.21.15.93:0/2400801732 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc6000781d0 con 0x7fc6140f6ca0 2021-11-20T09:00:37.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.756+0000 7fc618eb0700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc60405c2a0 0x7fc60405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:37.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.756+0000 7fc618eb0700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc60405c2a0 0x7fc60405e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc61405f7c0 tx=0x7fc608009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:37.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.760+0000 7fc6117fa700 1 -- 172.21.15.93:0/2400801732 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc60004aa80 con 0x7fc6140f6ca0 2021-11-20T09:00:37.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.941+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6140fde40 con 0x7fc6140f6ca0 2021-11-20T09:00:37.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.941+0000 7fc6117fa700 1 -- 172.21.15.93:0/2400801732 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7fc60004e0a0 con 0x7fc6140f6ca0 2021-11-20T09:00:37.941 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:37.941 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:37.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc60405c2a0 msgr2=0x7fc60405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:37.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc60405c2a0 0x7fc60405e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc61405f7c0 tx=0x7fc608009040).stop 2021-11-20T09:00:37.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f6ca0 msgr2=0x7fc61410bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:37.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f6ca0 0x7fc61410bd40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fc600007a60 tx=0x7fc600009b30).stop 2021-11-20T09:00:37.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 shutdown_connections 2021-11-20T09:00:37.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc60405c2a0 0x7fc60405e760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:37.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6140f5950 0x7fc6140f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:37.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 --2- 172.21.15.93:0/2400801732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140f6ca0 0x7fc61410bd40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:37.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 >> 172.21.15.93:0/2400801732 conn(0x7fc6140f04a0 msgr2=0x7fc6140fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:37.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 shutdown_connections 2021-11-20T09:00:37.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:37.943+0000 7fc61b114700 1 -- 172.21.15.93:0/2400801732 wait complete. 2021-11-20T09:00:37.946 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:37.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:37.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:38.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:38 smithi102 conmon[36472]: cluster 2021-11-20T09:00:37.578043+0000 mgr.smithi093.qdikkm (mgr.14162) 302 : cluster [DBG] pgmap v278: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:38.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:38 smithi102 conmon[36472]: audit 2021-11-20T09:00:37.942403+0000 mon.smithi093 (mon.0) 815 : audit [DBG] from='client.? 172.21.15.93:0/2400801732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:38.957 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:38.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:38 smithi093 conmon[33522]: cluster 2021-11-20T09:00:37.578043+0000 mgr.smithi093.qdikkm (mgr.14162) 302 : cluster [DBG] pgmap v278: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:38.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:38 smithi093 conmon[33522]: 2021-11-20T09:00:38.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:38 smithi093 conmon[33522]: audit 2021-11-20T09:00:37.942403+0000 mon.smithi093 (mon.0) 815 : audit [DBG] from='client.? 172.21.15.93:0/2400801732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:39.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 -- 172.21.15.93:0/2896975294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806a3b0 msgr2=0x7f423806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:39.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 --2- 172.21.15.93:0/2896975294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806a3b0 0x7f423806a830 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f422c004660 tx=0x7f422c009b30).stop 2021-11-20T09:00:39.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 -- 172.21.15.93:0/2896975294 shutdown_connections 2021-11-20T09:00:39.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 --2- 172.21.15.93:0/2896975294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806a3b0 0x7f423806a830 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:39.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 --2- 172.21.15.93:0/2896975294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423806b9c0 0x7f4238069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:39.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 -- 172.21.15.93:0/2896975294 >> 172.21.15.93:0/2896975294 conn(0x7f42380f1db0 msgr2=0x7f42380f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:39.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 -- 172.21.15.93:0/2896975294 shutdown_connections 2021-11-20T09:00:39.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 -- 172.21.15.93:0/2896975294 wait complete. 2021-11-20T09:00:39.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.083+0000 7f423fde9700 1 Processor -- start 2021-11-20T09:00:39.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423fde9700 1 -- start start 2021-11-20T09:00:39.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423fde9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423806a3b0 0x7f423810d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:39.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423fde9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806b9c0 0x7f423810d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:39.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423fde9700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423810df00 con 0x7f423806b9c0 2021-11-20T09:00:39.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423fde9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423810e040 con 0x7f423806a3b0 2021-11-20T09:00:39.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423d384700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806b9c0 0x7f423810d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:39.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423d384700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806b9c0 0x7f423810d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44388/0 (socket says 172.21.15.93:44388) 2021-11-20T09:00:39.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423d384700 1 -- 172.21.15.93:0/2940073760 learned_addr learned my addr 172.21.15.93:0/2940073760 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:39.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423d384700 1 -- 172.21.15.93:0/2940073760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423806a3b0 msgr2=0x7f423810d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:39.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423d384700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423806a3b0 0x7f423810d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:39.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423d384700 1 -- 172.21.15.93:0/2940073760 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c005040 con 0x7f423806b9c0 2021-11-20T09:00:39.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423d384700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806b9c0 0x7f423810d950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f422c009830 tx=0x7f422c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:39.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f42367fc700 1 -- 172.21.15.93:0/2940073760 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f422c018070 con 0x7f423806b9c0 2021-11-20T09:00:39.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f42367fc700 1 -- 172.21.15.93:0/2940073760 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f422c013410 con 0x7f423806b9c0 2021-11-20T09:00:39.092 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4238112a80 con 0x7f423806b9c0 2021-11-20T09:00:39.092 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f42367fc700 1 -- 172.21.15.93:0/2940073760 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f422c01c620 con 0x7f423806b9c0 2021-11-20T09:00:39.093 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.084+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4238113040 con 0x7f423806b9c0 2021-11-20T09:00:39.093 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.085+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4238027a00 con 0x7f423806b9c0 2021-11-20T09:00:39.094 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.086+0000 7f42367fc700 1 -- 172.21.15.93:0/2940073760 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f422c01c8e0 con 0x7f423806b9c0 2021-11-20T09:00:39.094 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.086+0000 7f42367fc700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f422005c2a0 0x7f422005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:39.095 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.086+0000 7f42367fc700 1 -- 172.21.15.93:0/2940073760 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f422c078200 con 0x7f423806b9c0 2021-11-20T09:00:39.095 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.086+0000 7f423db85700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f422005c2a0 0x7f422005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:39.095 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.086+0000 7f423db85700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f422005c2a0 0x7f422005e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4228007a60 tx=0x7f42280049c0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:39.096 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.090+0000 7f42367fc700 1 -- 172.21.15.93:0/2940073760 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f422c04aa00 con 0x7f423806b9c0 2021-11-20T09:00:39.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.277+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4238112c10 con 0x7f423806b9c0 2021-11-20T09:00:39.277 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.278+0000 7f42367fc700 1 -- 172.21.15.93:0/2940073760 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7f4238112c10 con 0x7f423806b9c0 2021-11-20T09:00:39.277 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:39.277 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:39.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f422005c2a0 msgr2=0x7f422005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:39.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f422005c2a0 0x7f422005e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4228007a60 tx=0x7f42280049c0).stop 2021-11-20T09:00:39.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806b9c0 msgr2=0x7f423810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:39.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806b9c0 0x7f423810d950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f422c009830 tx=0x7f422c009b30).stop 2021-11-20T09:00:39.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 shutdown_connections 2021-11-20T09:00:39.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f422005c2a0 0x7f422005e760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:39.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423806a3b0 0x7f423810d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:39.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 --2- 172.21.15.93:0/2940073760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f423806b9c0 0x7f423810d950 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:39.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 >> 172.21.15.93:0/2940073760 conn(0x7f42380f1db0 msgr2=0x7f42380f8850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:39.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 shutdown_connections 2021-11-20T09:00:39.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:39.279+0000 7f423fde9700 1 -- 172.21.15.93:0/2940073760 wait complete. 2021-11-20T09:00:39.282 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:39.290 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:39.290 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:39.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:39 smithi102 conmon[36472]: audit 2021-11-20T09:00:39.278674+0000 mon.smithi093 (mon.0) 816 : audit [DBG] from='client.? 172.21.15.93:0/2940073760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:39.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:39 smithi093 conmon[33522]: audit 2021-11-20T09:00:39.278674+0000 mon.smithi093 2021-11-20T09:00:39.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:39 smithi093 conmon[33522]: (mon.0) 816 : audit [DBG] from='client.? 172.21.15.93:0/2940073760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:40.292 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:40.418 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- 172.21.15.93:0/3729846425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 msgr2=0x7f984c0f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:40.419 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 --2- 172.21.15.93:0/3729846425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c0f6ac0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f983c004660 tx=0x7f983c009b30).stop 2021-11-20T09:00:40.419 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- 172.21.15.93:0/3729846425 shutdown_connections 2021-11-20T09:00:40.419 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 --2- 172.21.15.93:0/3729846425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f984c0f7940 0x7f984c0f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:40.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 --2- 172.21.15.93:0/3729846425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c0f6ac0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:40.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- 172.21.15.93:0/3729846425 >> 172.21.15.93:0/3729846425 conn(0x7f984c0f1e00 msgr2=0x7f984c0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:40.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- 172.21.15.93:0/3729846425 shutdown_connections 2021-11-20T09:00:40.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- 172.21.15.93:0/3729846425 wait complete. 2021-11-20T09:00:40.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 Processor -- start 2021-11-20T09:00:40.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- start start 2021-11-20T09:00:40.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:40.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f984c0f7940 0x7f984c107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:40.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c107990 con 0x7f984c0f66a0 2021-11-20T09:00:40.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.418+0000 7f9852395700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c107ad0 con 0x7f984c0f7940 2021-11-20T09:00:40.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:40.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44396/0 (socket says 172.21.15.93:44396) 2021-11-20T09:00:40.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984bfff700 1 -- 172.21.15.93:0/2339758509 learned_addr learned my addr 172.21.15.93:0/2339758509 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:40.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984b7fe700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f984c0f7940 0x7f984c107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:40.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984bfff700 1 -- 172.21.15.93:0/2339758509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f984c0f7940 msgr2=0x7f984c107350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:40.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984bfff700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f984c0f7940 0x7f984c107350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:40.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984bfff700 1 -- 172.21.15.93:0/2339758509 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c005040 con 0x7f984c0f66a0 2021-11-20T09:00:40.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f984bfff700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c106e60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f983c009f50 tx=0x7f983c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:40.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f9848ff9700 1 -- 172.21.15.93:0/2339758509 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f983c018070 con 0x7f984c0f66a0 2021-11-20T09:00:40.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f9848ff9700 1 -- 172.21.15.93:0/2339758509 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f983c013410 con 0x7f984c0f66a0 2021-11-20T09:00:40.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f9848ff9700 1 -- 172.21.15.93:0/2339758509 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f983c01c620 con 0x7f984c0f66a0 2021-11-20T09:00:40.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984c10c510 con 0x7f984c0f66a0 2021-11-20T09:00:40.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.419+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984c10ca70 con 0x7f984c0f66a0 2021-11-20T09:00:40.425 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.420+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984c027a00 con 0x7f984c0f66a0 2021-11-20T09:00:40.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.420+0000 7f9848ff9700 1 -- 172.21.15.93:0/2339758509 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f983c01d420 con 0x7f984c0f66a0 2021-11-20T09:00:40.427 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.421+0000 7f9848ff9700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f983405c2a0 0x7f983405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:40.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.421+0000 7f9848ff9700 1 -- 172.21.15.93:0/2339758509 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f983c0781d0 con 0x7f984c0f66a0 2021-11-20T09:00:40.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.421+0000 7f984b7fe700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f983405c2a0 0x7f983405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:40.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.421+0000 7f984b7fe700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f983405c2a0 0x7f983405e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f984000b790 tx=0x7f9840008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:40.428 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.425+0000 7f9848ff9700 1 -- 172.21.15.93:0/2339758509 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f983c04a9d0 con 0x7f984c0f66a0 2021-11-20T09:00:40.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.609+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f984c0feeb0 con 0x7f984c0f66a0 2021-11-20T09:00:40.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.609+0000 7f9848ff9700 1 -- 172.21.15.93:0/2339758509 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7f983c04dff0 con 0x7f984c0f66a0 2021-11-20T09:00:40.609 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:40.609 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:40.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f983405c2a0 msgr2=0x7f983405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:40.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f983405c2a0 0x7f983405e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f984000b790 tx=0x7f9840008040).stop 2021-11-20T09:00:40.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 msgr2=0x7f984c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:40.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c106e60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f983c009f50 tx=0x7f983c009b30).stop 2021-11-20T09:00:40.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 shutdown_connections 2021-11-20T09:00:40.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f983405c2a0 0x7f983405e760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:40.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f984c0f66a0 0x7f984c106e60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:40.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 --2- 172.21.15.93:0/2339758509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f984c0f7940 0x7f984c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:40.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 >> 172.21.15.93:0/2339758509 conn(0x7f984c0f1e00 msgr2=0x7f984c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:40.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 shutdown_connections 2021-11-20T09:00:40.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:40.611+0000 7f9852395700 1 -- 172.21.15.93:0/2339758509 wait complete. 2021-11-20T09:00:40.614 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:40.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:40.623 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:40.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:40 smithi102 conmon[36472]: cluster 2021-11-20T09:00:39.578554+0000 mgr.smithi093.qdikkm (mgr.14162) 303 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 201 B/s wr, 1 op/s 2021-11-20T09:00:40.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:40 smithi102 conmon[36472]: audit 2021-11-20T09:00:40.610385+0000 mon.smithi093 (mon.0) 817 : audit [DBG] from='client.? 172.21.15.93:0/2339758509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:40.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:40 smithi093 conmon[33522]: cluster 2021-11-20T09:00:39.578554+0000 mgr.smithi093.qdikkm 2021-11-20T09:00:40.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:40 smithi093 conmon[33522]: (mgr.14162) 303 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 201 B/s wr, 1 op/s 2021-11-20T09:00:40.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:40 smithi093 conmon[33522]: audit 2021-11-20T09:00:40.610385+0000 2021-11-20T09:00:40.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:40 smithi093 conmon[33522]: mon.smithi093 (mon.0) 817 : audit [DBG] from='client.? 172.21.15.93:0/2339758509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:41.624 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:41.758 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.749+0000 7fc8cedec700 1 -- 172.21.15.93:0/3193688806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 msgr2=0x7fc8c80f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:41.759 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.749+0000 7fc8cedec700 1 --2- 172.21.15.93:0/3193688806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c80f69f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fc8c4004660 tx=0x7fc8c4009b30).stop 2021-11-20T09:00:41.759 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.749+0000 7fc8cedec700 1 -- 172.21.15.93:0/3193688806 shutdown_connections 2021-11-20T09:00:41.759 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.749+0000 7fc8cedec700 1 --2- 172.21.15.93:0/3193688806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c80f7870 0x7fc8c80f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:41.760 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.749+0000 7fc8cedec700 1 --2- 172.21.15.93:0/3193688806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c80f69f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:41.760 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.749+0000 7fc8cedec700 1 -- 172.21.15.93:0/3193688806 >> 172.21.15.93:0/3193688806 conn(0x7fc8c80f1d10 msgr2=0x7fc8c80f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:41.761 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 -- 172.21.15.93:0/3193688806 shutdown_connections 2021-11-20T09:00:41.761 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 -- 172.21.15.93:0/3193688806 wait complete. 2021-11-20T09:00:41.762 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 Processor -- start 2021-11-20T09:00:41.762 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 -- start start 2021-11-20T09:00:41.762 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c8106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:41.763 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c80f7870 0x7fc8c8107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:41.763 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c8107890 con 0x7fc8c80f65d0 2021-11-20T09:00:41.764 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cedec700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c81080d0 con 0x7fc8c80f7870 2021-11-20T09:00:41.764 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cddea700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c8106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:41.765 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cddea700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c8106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44408/0 (socket says 172.21.15.93:44408) 2021-11-20T09:00:41.765 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cddea700 1 -- 172.21.15.93:0/2011424295 learned_addr learned my addr 172.21.15.93:0/2011424295 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:41.765 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.750+0000 7fc8cd5e9700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c80f7870 0x7fc8c8107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:41.766 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8cddea700 1 -- 172.21.15.93:0/2011424295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c80f7870 msgr2=0x7fc8c8107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:41.766 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8cddea700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c80f7870 0x7fc8c8107290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:41.767 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8cddea700 1 -- 172.21.15.93:0/2011424295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c4005040 con 0x7fc8c80f65d0 2021-11-20T09:00:41.767 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8cddea700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c8106da0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc8c4009f50 tx=0x7fc8c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:41.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8be7fc700 1 -- 172.21.15.93:0/2011424295 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8c4018070 con 0x7fc8c80f65d0 2021-11-20T09:00:41.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8be7fc700 1 -- 172.21.15.93:0/2011424295 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8c4013410 con 0x7fc8c80f65d0 2021-11-20T09:00:41.768 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c8108350 con 0x7fc8c80f65d0 2021-11-20T09:00:41.769 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8be7fc700 1 -- 172.21.15.93:0/2011424295 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8c401c620 con 0x7fc8c80f65d0 2021-11-20T09:00:41.769 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.751+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c8108830 con 0x7fc8c80f65d0 2021-11-20T09:00:41.770 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.752+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c80f59c0 con 0x7fc8c80f65d0 2021-11-20T09:00:41.770 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.752+0000 7fc8be7fc700 1 -- 172.21.15.93:0/2011424295 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc8c401d420 con 0x7fc8c80f65d0 2021-11-20T09:00:41.771 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.752+0000 7fc8be7fc700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc8b405c300 0x7fc8b405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:41.771 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.753+0000 7fc8cd5e9700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc8b405c300 0x7fc8b405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:41.771 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.753+0000 7fc8be7fc700 1 -- 172.21.15.93:0/2011424295 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc8c40780f0 con 0x7fc8c80f65d0 2021-11-20T09:00:41.772 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.753+0000 7fc8cd5e9700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc8b405c300 0x7fc8b405e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fc8b8007fd0 tx=0x7fc8b8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:41.772 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.756+0000 7fc8be7fc700 1 -- 172.21.15.93:0/2011424295 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8c404a9d0 con 0x7fc8c80f65d0 2021-11-20T09:00:41.937 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.938+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8c80fecf0 con 0x7fc8c80f65d0 2021-11-20T09:00:41.938 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.938+0000 7fc8be7fc700 1 -- 172.21.15.93:0/2011424295 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7fc8c404dff0 con 0x7fc8c80f65d0 2021-11-20T09:00:41.939 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:41.939 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:41.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc8b405c300 msgr2=0x7fc8b405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:41.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc8b405c300 0x7fc8b405e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fc8b8007fd0 tx=0x7fc8b8009040).stop 2021-11-20T09:00:41.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 msgr2=0x7fc8c8106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:41.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c8106da0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc8c4009f50 tx=0x7fc8c4009b30).stop 2021-11-20T09:00:41.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 shutdown_connections 2021-11-20T09:00:41.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc8b405c300 0x7fc8b405e7c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:41.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8c80f65d0 0x7fc8c8106da0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:41.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 --2- 172.21.15.93:0/2011424295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8c80f7870 0x7fc8c8107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:41.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 >> 172.21.15.93:0/2011424295 conn(0x7fc8c80f1d10 msgr2=0x7fc8c80fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:41.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 shutdown_connections 2021-11-20T09:00:41.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:41.940+0000 7fc8cedec700 1 -- 172.21.15.93:0/2011424295 wait complete. 2021-11-20T09:00:41.948 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:41.952 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:41.953 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:42.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:42 smithi102 conmon[36472]: cluster 2021-11-20T09:00:41.579086+0000 mgr.smithi093.qdikkm (mgr.14162) 304 : cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:42.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:42 smithi102 conmon[36472]: audit 2021-11-20T09:00:41.939541 2021-11-20T09:00:42.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:42 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 818 : audit [DBG] from='client.? 172.21.15.93:0/2011424295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:42.955 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:42.960 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:42 smithi093 conmon[33522]: cluster 2021-11-20T09:00:41.579086+0000 mgr.smithi093.qdikkm (mgr.14162) 304 : cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:42.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:42 smithi093 conmon[33522]: audit 2021-11-20T09:00:41.939541+0000 mon.smithi093 (mon.0) 818 : audit [DBG] from='client.? 172.21.15.93:0/2011424295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:43.083 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 -- 172.21.15.93:0/42540034 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 msgr2=0x7f46940f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:43.084 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 --2- 172.21.15.93:0/42540034 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f46940f6c80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f4688004660 tx=0x7f4688009b30).stop 2021-11-20T09:00:43.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 -- 172.21.15.93:0/42540034 shutdown_connections 2021-11-20T09:00:43.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 --2- 172.21.15.93:0/42540034 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f46940f6c80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:43.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 --2- 172.21.15.93:0/42540034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46940f5580 0x7f46940f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:43.085 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 -- 172.21.15.93:0/42540034 >> 172.21.15.93:0/42540034 conn(0x7f46940f0ce0 msgr2=0x7f46940f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:43.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 -- 172.21.15.93:0/42540034 shutdown_connections 2021-11-20T09:00:43.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.080+0000 7f4698920700 1 -- 172.21.15.93:0/42540034 wait complete. 2021-11-20T09:00:43.086 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4698920700 1 Processor -- start 2021-11-20T09:00:43.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4698920700 1 -- start start 2021-11-20T09:00:43.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4698920700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46940f5580 0x7f4694060570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:43.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4698920700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f469410bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:43.087 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4698920700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469410c100 con 0x7f46940f6820 2021-11-20T09:00:43.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4698920700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469410c240 con 0x7f46940f5580 2021-11-20T09:00:43.088 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4691d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f469410bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:43.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4691d9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f469410bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44420/0 (socket says 172.21.15.93:44420) 2021-11-20T09:00:43.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4691d9b700 1 -- 172.21.15.93:0/2880102735 learned_addr learned my addr 172.21.15.93:0/2880102735 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:43.089 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f469259c700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46940f5580 0x7f4694060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:43.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4691d9b700 1 -- 172.21.15.93:0/2880102735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46940f5580 msgr2=0x7f4694060570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:43.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4691d9b700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46940f5580 0x7f4694060570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:43.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.081+0000 7f4691d9b700 1 -- 172.21.15.93:0/2880102735 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4688005040 con 0x7f46940f6820 2021-11-20T09:00:43.090 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.082+0000 7f4691d9b700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f469410bc10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f4688009830 tx=0x7f4688009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:43.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.082+0000 7f4682ffd700 1 -- 172.21.15.93:0/2880102735 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4688018070 con 0x7f46940f6820 2021-11-20T09:00:43.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.082+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469410c440 con 0x7f46940f6820 2021-11-20T09:00:43.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.082+0000 7f4682ffd700 1 -- 172.21.15.93:0/2880102735 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4688013410 con 0x7f46940f6820 2021-11-20T09:00:43.091 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.082+0000 7f4682ffd700 1 -- 172.21.15.93:0/2880102735 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f468801c3c0 con 0x7f46940f6820 2021-11-20T09:00:43.092 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.082+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469410c9a0 con 0x7f46940f6820 2021-11-20T09:00:43.092 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.083+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4694027a00 con 0x7f46940f6820 2021-11-20T09:00:43.092 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.083+0000 7f4682ffd700 1 -- 172.21.15.93:0/2880102735 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f468801d420 con 0x7f46940f6820 2021-11-20T09:00:43.092 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.083+0000 7f4682ffd700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f467c05c2a0 0x7f467c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:43.093 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.083+0000 7f4682ffd700 1 -- 172.21.15.93:0/2880102735 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4688077fa0 con 0x7f46940f6820 2021-11-20T09:00:43.093 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.083+0000 7f469259c700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f467c05c2a0 0x7f467c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:43.093 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.084+0000 7f469259c700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f467c05c2a0 0x7f467c05e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f46940f6680 tx=0x7f4684009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:43.094 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.087+0000 7f4682ffd700 1 -- 172.21.15.93:0/2880102735 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f468804a800 con 0x7f46940f6820 2021-11-20T09:00:43.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.275+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46940fdd10 con 0x7f46940f6820 2021-11-20T09:00:43.282 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.276+0000 7f4682ffd700 1 -- 172.21.15.93:0/2880102735 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2189 (secure 0 0 0) 0x7f468804de20 con 0x7f46940f6820 2021-11-20T09:00:43.283 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:43.283 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:32.631495+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24341":{"gid":24341,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":75,"state":"up:clientreplay","state_seq":52,"addr":"172.21.15.102:6825/1330867096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1330867096},{"type":"v1","addr":"172.21.15.102:6825","nonce":1330867096}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:43.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.277+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f467c05c2a0 msgr2=0x7f467c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:43.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.277+0000 7f4698920700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f467c05c2a0 0x7f467c05e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f46940f6680 tx=0x7f4684009040).stop 2021-11-20T09:00:43.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.277+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 msgr2=0x7f469410bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:43.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.277+0000 7f4698920700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f469410bc10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f4688009830 tx=0x7f4688009b30).stop 2021-11-20T09:00:43.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.278+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 shutdown_connections 2021-11-20T09:00:43.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.278+0000 7f4698920700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f467c05c2a0 0x7f467c05e760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:43.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.278+0000 7f4698920700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46940f5580 0x7f4694060570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:43.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.278+0000 7f4698920700 1 --2- 172.21.15.93:0/2880102735 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46940f6820 0x7f469410bc10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:43.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.278+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 >> 172.21.15.93:0/2880102735 conn(0x7f46940f0ce0 msgr2=0x7f46940fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:43.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.278+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 shutdown_connections 2021-11-20T09:00:43.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:43.278+0000 7f4698920700 1 -- 172.21.15.93:0/2880102735 wait complete. 2021-11-20T09:00:43.292 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 79 2021-11-20T09:00:43.293 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:32.631495+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24341': {'gid': 24341, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 75, 'state': 'up:clientreplay', 'state_seq': 52, 'addr': '172.21.15.102:6825/1330867096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 1330867096}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 1330867096}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:43.293 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24341:up:clientreplay 2021-11-20T09:00:43.917 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:43 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.155666+0000 mon.smithi093 (mon.0) 819 : cluster [INF] daemon mds.cephfs.smithi102.lhrlmh restarted 2021-11-20T09:00:43.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:43 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.159927+0000 mon.smithi093 (mon.0) 820 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T09:00:43.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:43 smithi102 conmon[36472]: audit 2021-11-20 2021-11-20T09:00:43.924 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:43 smithi102 conmon[36472]: T09:00:43.276924+0000 mon.smithi093 (mon.0) 821 : audit [DBG] from='client.? 172.21.15.93:0/2880102735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:43.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:43 smithi093 conmon[33522]: debug 2021-11-20T09:00:43.662+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:00:43.976 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:43 smithi093 conmon[33522]: cluster 2021-11-20T09:00: 2021-11-20T09:00:43.977 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:43 smithi093 conmon[33522]: 43.155666+0000 mon.smithi093 (mon.0) 819 : cluster [INF] daemon mds.cephfs.smithi102.lhrlmh restarted 2021-11-20T09:00:43.989 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:43 smithi093 conmon[33522]: cluster 2021-11-20T09:00:43. 2021-11-20T09:00:43.989 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:43 smithi093 conmon[33522]: 159927+0000 mon.smithi093 (mon.0) 820 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T09:00:43.990 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:43 smithi093 conmon[33522]: audit 2021-11-20T09:00:43.276924 2021-11-20T09:00:43.990 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:43 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 821 : audit [DBG] from='client.? 172.21.15.93:0/2880102735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:44.295 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:44.442 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 -- 172.21.15.93:0/2609282362 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 msgr2=0x7f13880f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:44.442 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 --2- 172.21.15.93:0/2609282362 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f13880f59a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f1384004660 tx=0x7f1384009b30).stop 2021-11-20T09:00:44.442 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 -- 172.21.15.93:0/2609282362 shutdown_connections 2021-11-20T09:00:44.443 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 --2- 172.21.15.93:0/2609282362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13880f6820 0x7f13880f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:44.443 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 --2- 172.21.15.93:0/2609282362 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f13880f59a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:44.443 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 -- 172.21.15.93:0/2609282362 >> 172.21.15.93:0/2609282362 conn(0x7f13880f0ce0 msgr2=0x7f13880f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:44.443 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 -- 172.21.15.93:0/2609282362 shutdown_connections 2021-11-20T09:00:44.444 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 -- 172.21.15.93:0/2609282362 wait complete. 2021-11-20T09:00:44.444 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.432+0000 7f139031b700 1 Processor -- start 2021-11-20T09:00:44.444 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f139031b700 1 -- start start 2021-11-20T09:00:44.444 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f139031b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f1388060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:44.444 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f139031b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13880f6820 0x7f138810bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:44.445 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f139031b700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138810c1b0 con 0x7f13880f5580 2021-11-20T09:00:44.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f139031b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138810c2f0 con 0x7f13880f6820 2021-11-20T09:00:44.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138e0b7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f1388060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:44.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138e0b7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f1388060530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44428/0 (socket says 172.21.15.93:44428) 2021-11-20T09:00:44.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138e0b7700 1 -- 172.21.15.93:0/3227125668 learned_addr learned my addr 172.21.15.93:0/3227125668 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:44.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138d8b6700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13880f6820 0x7f138810bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:44.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138e0b7700 1 -- 172.21.15.93:0/3227125668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13880f6820 msgr2=0x7f138810bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:44.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138e0b7700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13880f6820 0x7f138810bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:44.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138e0b7700 1 -- 172.21.15.93:0/3227125668 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384005040 con 0x7f13880f5580 2021-11-20T09:00:44.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138d8b6700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13880f6820 0x7f138810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:00:44.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f138e0b7700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f1388060530 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1384004e30 tx=0x7f1384009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:44.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f137effd700 1 -- 172.21.15.93:0/3227125668 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1384018070 con 0x7f13880f5580 2021-11-20T09:00:44.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f137effd700 1 -- 172.21.15.93:0/3227125668 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1384013410 con 0x7f13880f5580 2021-11-20T09:00:44.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138810c4f0 con 0x7f13880f5580 2021-11-20T09:00:44.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f137effd700 1 -- 172.21.15.93:0/3227125668 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f138401c3c0 con 0x7f13880f5580 2021-11-20T09:00:44.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.433+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138810ca20 con 0x7f13880f5580 2021-11-20T09:00:44.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.434+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1388027a00 con 0x7f13880f5580 2021-11-20T09:00:44.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.435+0000 7f137effd700 1 -- 172.21.15.93:0/3227125668 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f138401d420 con 0x7f13880f5580 2021-11-20T09:00:44.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.435+0000 7f137effd700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f137405c2a0 0x7f137405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:44.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.435+0000 7f137effd700 1 -- 172.21.15.93:0/3227125668 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1384078010 con 0x7f13880f5580 2021-11-20T09:00:44.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.435+0000 7f138d8b6700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f137405c2a0 0x7f137405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:44.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.436+0000 7f138d8b6700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f137405c2a0 0x7f137405e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1378009fd0 tx=0x7f137800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:44.454 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.439+0000 7f137effd700 1 -- 172.21.15.93:0/3227125668 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f138404a820 con 0x7f13880f5580 2021-11-20T09:00:44.627 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.623+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13880fdd10 con 0x7f13880f5580 2021-11-20T09:00:44.627 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.624+0000 7f137effd700 1 -- 172.21.15.93:0/3227125668 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2183 (secure 0 0 0) 0x7f138404de40 con 0x7f13880f5580 2021-11-20T09:00:44.630 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:44.631 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:43.669364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:replay","state_seq":3,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:44.634 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f137405c2a0 msgr2=0x7f137405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:44.634 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f137405c2a0 0x7f137405e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1378009fd0 tx=0x7f137800b040).stop 2021-11-20T09:00:44.635 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 msgr2=0x7f1388060530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:44.636 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f1388060530 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f1384004e30 tx=0x7f1384009b30).stop 2021-11-20T09:00:44.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 shutdown_connections 2021-11-20T09:00:44.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f137405c2a0 0x7f137405e760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:44.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13880f5580 0x7f1388060530 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:44.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 --2- 172.21.15.93:0/3227125668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13880f6820 0x7f138810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:44.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.625+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 >> 172.21.15.93:0/3227125668 conn(0x7f13880f0ce0 msgr2=0x7f13880fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:44.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.626+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 shutdown_connections 2021-11-20T09:00:44.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:44.626+0000 7f139031b700 1 -- 172.21.15.93:0/3227125668 wait complete. 2021-11-20T09:00:44.643 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 81 2021-11-20T09:00:44.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:43.669364+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:44.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:replay 2021-11-20T09:00:44.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.579546+0000 mgr.smithi093.qdikkm (mgr.14162) 305 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2021-11-20T09:00:44.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.663541+0000 2021-11-20T09:00:44.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: mon.smithi093 (mon.0) 822 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:00:44.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.669284+0000 mon.smithi093 (mon.0) 823 : cluster [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:boot 2021-11-20T09:00:44.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.669331+0000 mon.smithi093 (mon.0) 824 : cluster [INF] Standby daemon mds.cephfs.smithi102.hjodmi assigned to filesystem cephfs as rank 0 2021-11-20T09:00:44.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.669413+0000 mon.smithi093 (mon.0) 825 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:00:44.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.669600+0000 2021-11-20T09:00:44.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: mon.smithi093 (mon.0) 826 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:00:44.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: audit 2021-11-20T09:00:43.669714+0000 2021-11-20T09:00:44.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: mon.smithi093 (mon.0) 827 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T09:00:44.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: cluster 2021-11-20T09:00:43.674438+0000 mon.smithi093 2021-11-20T09:00:44.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: (mon.0) 828 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:replay} 1 up:standby 2021-11-20T09:00:44.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: audit 2021-11-20T09:00:44.624632+0000 mon.smithi093 2021-11-20T09:00:44.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:44 smithi102 conmon[36472]: (mon.0) 829 : audit [DBG] from='client.? 172.21.15.93:0/3227125668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:44.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster 2021-11-20T09:00:43.579546+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T09:00:44.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: ) 305 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2021-11-20T09:00:44.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster 2021-11-20T09:00:43.663541+0000 mon.smithi093 (mon.0) 822 : cluster 2021-11-20T09:00:44.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:00:44.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster 2021-11-20T09 2021-11-20T09:00:44.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: :00:43.669284+0000 mon.smithi093 (mon.0) 823 : cluster [DBG] 2021-11-20T09:00:44.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:boot 2021-11-20T09:00:44.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster 2021-11-20T09:00: 2021-11-20T09:00:44.964 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 43.669331+0000 mon.smithi093 (mon.0) 824 : 2021-11-20T09:00:44.965 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster [INF] Standby daemon mds.cephfs.smithi102.hjodmi assigned to filesystem cephfs as rank 0 2021-11-20T09:00:44.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster 2021-11-20T09: 2021-11-20T09:00:44.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 00:43.669413+0000 mon.smithi093 (mon.0 2021-11-20T09:00:44.966 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: ) 825 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:00:44.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T09:00:44.967 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 20T09:00:43.669600+0000 mon.smithi093 2021-11-20T09:00:44.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: (mon.0) 826 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:00:44.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 2021-11-20T09:00:44.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: audit 2021-11-20T09:00:44.968 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 2021-11-20T09:00:43.669714+0000 mon.smithi093 ( 2021-11-20T09:00:44.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: mon.0) 827 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T09:00:44.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: cluster 2021- 2021-11-20T09:00:44.969 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 11-20T09:00:43.674438+0000 2021-11-20T09:00:44.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: mon.smithi093 (mon.0) 828 : cluster [DBG] 2021-11-20T09:00:44.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:replay} 1 up:standby 2021-11-20T09:00:44.970 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: audit 2021-11-20T09:00:44.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 2021-11-20T09:00:44.624632+0000 2021-11-20T09:00:44.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: mon.smithi093 (mon.0) 829 : audit [DBG] from='client.? 172.21.15.93:0/3227125668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:44.971 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:44 smithi093 conmon[33522]: 2021-11-20T09:00:45.659 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:45.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- 172.21.15.93:0/3721906080 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 msgr2=0x7f440c0f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:45.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 --2- 172.21.15.93:0/3721906080 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0f6cb0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f43fc004660 tx=0x7f43fc009b30).stop 2021-11-20T09:00:45.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- 172.21.15.93:0/3721906080 shutdown_connections 2021-11-20T09:00:45.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 --2- 172.21.15.93:0/3721906080 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0f6cb0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:45.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 --2- 172.21.15.93:0/3721906080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f440c0f55b0 0x7f440c0f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:45.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- 172.21.15.93:0/3721906080 >> 172.21.15.93:0/3721906080 conn(0x7f440c0f0d10 msgr2=0x7f440c0f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:45.801 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- 172.21.15.93:0/3721906080 shutdown_connections 2021-11-20T09:00:45.801 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- 172.21.15.93:0/3721906080 wait complete. 2021-11-20T09:00:45.801 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 Processor -- start 2021-11-20T09:00:45.801 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- start start 2021-11-20T09:00:45.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f440c0f55b0 0x7f440c0fd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:45.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0fbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:45.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c0fdb00 con 0x7f440c0f6850 2021-11-20T09:00:45.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f44140b1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c0fc0a0 con 0x7f440c0f55b0 2021-11-20T09:00:45.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f441164c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0fbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:45.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f441164c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0fbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44442/0 (socket says 172.21.15.93:44442) 2021-11-20T09:00:45.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.793+0000 7f441164c700 1 -- 172.21.15.93:0/1645199738 learned_addr learned my addr 172.21.15.93:0/1645199738 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:45.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f4411e4d700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f440c0f55b0 0x7f440c0fd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:45.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f441164c700 1 -- 172.21.15.93:0/1645199738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f440c0f55b0 msgr2=0x7f440c0fd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:45.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f441164c700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f440c0f55b0 0x7f440c0fd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:45.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f441164c700 1 -- 172.21.15.93:0/1645199738 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc005040 con 0x7f440c0f6850 2021-11-20T09:00:45.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f4411e4d700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f440c0f55b0 0x7f440c0fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:00:45.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f441164c700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0fbbb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f43fc000c00 tx=0x7f43fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:45.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f44027fc700 1 -- 172.21.15.93:0/1645199738 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43fc019070 con 0x7f440c0f6850 2021-11-20T09:00:45.807 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f44027fc700 1 -- 172.21.15.93:0/1645199738 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43fc007e70 con 0x7f440c0f6850 2021-11-20T09:00:45.808 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f44027fc700 1 -- 172.21.15.93:0/1645199738 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43fc014d00 con 0x7f440c0f6850 2021-11-20T09:00:45.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c0fc2c0 con 0x7f440c0f6850 2021-11-20T09:00:45.809 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.794+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c0fc820 con 0x7f440c0f6850 2021-11-20T09:00:45.810 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.795+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440c046330 con 0x7f440c0f6850 2021-11-20T09:00:45.810 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.795+0000 7f44027fc700 1 -- 172.21.15.93:0/1645199738 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f43fc025070 con 0x7f440c0f6850 2021-11-20T09:00:45.811 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.795+0000 7f44027fc700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f43f805c2a0 0x7f43f805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:45.811 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.796+0000 7f44027fc700 1 -- 172.21.15.93:0/1645199738 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f43fc0773d0 con 0x7f440c0f6850 2021-11-20T09:00:45.811 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.796+0000 7f4411e4d700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f43f805c2a0 0x7f43f805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:45.812 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.796+0000 7f4411e4d700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f43f805c2a0 0x7f43f805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4408004fd0 tx=0x7f440800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:45.812 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.799+0000 7f44027fc700 1 -- 172.21.15.93:0/1645199738 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43fc049be0 con 0x7f440c0f6850 2021-11-20T09:00:45.973 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:45 smithi093 conmon[33522]: cluster 2021-11-20T09:00:44.678748+0000 mon.smithi093 (mon.0) 830 : cluster 2021-11-20T09:00:45.974 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:45 smithi093 conmon[33522]: [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:reconnect 2021-11-20T09:00:45.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:45 smithi093 conmon[33522]: cluster 2021-11-20T09:00:44.678813+0000 mon.smithi093 (mon.0) 831 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:reconnect} 1 up:standby 2021-11-20T09:00:45.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:45 smithi093 conmon[33522]: cluster 2021-11-20T09:00 2021-11-20T09:00:45.975 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:45 smithi093 conmon[33522]: :44.699236+0000 mds.cephfs.smithi102.hjodmi (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:00:45.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.982+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f440c027a00 con 0x7f440c0f6850 2021-11-20T09:00:45.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.983+0000 7f44027fc700 1 -- 172.21.15.93:0/1645199738 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2183 (secure 0 0 0) 0x7f43fc04d200 con 0x7f440c0f6850 2021-11-20T09:00:45.990 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:45.990 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:45.677819+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:45.999 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f43f805c2a0 msgr2=0x7f43f805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:46.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f43f805c2a0 0x7f43f805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4408004fd0 tx=0x7f440800b040).stop 2021-11-20T09:00:46.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 msgr2=0x7f440c0fbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:46.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0fbbb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f43fc000c00 tx=0x7f43fc00b040).stop 2021-11-20T09:00:46.000 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 shutdown_connections 2021-11-20T09:00:46.001 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f43f805c2a0 0x7f43f805e760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:46.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f440c0f55b0 0x7f440c0fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:46.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 --2- 172.21.15.93:0/1645199738 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f440c0f6850 0x7f440c0fbbb0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:46.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 >> 172.21.15.93:0/1645199738 conn(0x7f440c0f0d10 msgr2=0x7f440c0fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:46.009 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 shutdown_connections 2021-11-20T09:00:46.010 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:45.984+0000 7f44140b1700 1 -- 172.21.15.93:0/1645199738 wait complete. 2021-11-20T09:00:46.010 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 83 2021-11-20T09:00:46.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:45.677819+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:46.011 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:rejoin 2021-11-20T09:00:46.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:45 smithi102 conmon[36472]: cluster 2021-11-20T09:00:44.678748+0000 mon.smithi093 (mon.0 2021-11-20T09:00:46.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:45 smithi102 conmon[36472]: ) 830 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:reconnect 2021-11-20T09:00:46.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:45 smithi102 conmon[36472]: cluster 2021-11-20T09:00:44.678813+0000 mon.smithi093 (mon.0) 831 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:reconnect} 1 up:standby 2021-11-20T09:00:46.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:45 smithi102 conmon[36472]: cluster 2021-11-20T09:00:44.699236+0000 mds.cephfs.smithi102.hjodmi 2021-11-20T09:00:46.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:45 smithi102 conmon[36472]: (mds.0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:00:46.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: cluster 2021-11-20T09:00:45.580203+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T09:00:46.922 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: 14162) 306 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2021-11-20T09:00:46.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: cluster 2021-11-20T09:00:45.684494+0000 mon.smithi093 (mon.0) 832 : cluster 2021-11-20T09:00:46.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:rejoin 2021-11-20T09:00:46.925 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: cluster 2021- 2021-11-20T09:00:46.926 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: 11-20T09:00:45.684568+0000 mon.smithi093 (mon.0) 833 : cluster 2021-11-20T09:00:46.943 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:rejoin} 1 up:standby 2021-11-20T09:00:46.944 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: audit 2021-11-20T09: 2021-11-20T09:00:46.944 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: 00:45.983737+0000 mon.smithi093 (mon.0) 834 2021-11-20T09:00:46.945 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:46 smithi102 conmon[36472]: : audit [DBG] from='client.? 172.21.15.93:0/1645199738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:46.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: cluster 2021-11-20T09:00:45.580203+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T09:00:46.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: ) 306 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 3.3 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2021-11-20T09:00:46.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: cluster 2021-11-20T09:00:45.684494+0000 mon.smithi093 (mon.0) 832 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:rejoin 2021-11-20T09:00:46.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: cluster 2021-11-20T09:00:45.684568+0000 mon.smithi093 (mon.0) 833 : cluster 2021-11-20T09:00:46.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:rejoin} 1 up:standby 2021-11-20T09:00:46.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: audit 2021-11-20T09:00:46.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: 2021-11-20T09:00: 2021-11-20T09:00:46.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: 45.983737+0000 mon.smithi093 (mon.0 2021-11-20T09:00:46.963 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:46 smithi093 conmon[33522]: ) 834 : audit [DBG] from='client.? 172.21.15.93:0/1645199738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:47.027 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:47.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2118978859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 msgr2=0x7fa1e40f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:47.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2118978859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40f5990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fa1d8004660 tx=0x7fa1d8009b30).stop 2021-11-20T09:00:47.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2118978859 shutdown_connections 2021-11-20T09:00:47.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2118978859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1e40f6810 0x7fa1e40f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:47.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2118978859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40f5990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:47.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2118978859 >> 172.21.15.93:0/2118978859 conn(0x7fa1e40f0cb0 msgr2=0x7fa1e40f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:47.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2118978859 shutdown_connections 2021-11-20T09:00:47.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.165+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2118978859 wait complete. 2021-11-20T09:00:47.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1eacf2700 1 Processor -- start 2021-11-20T09:00:47.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1eacf2700 1 -- start start 2021-11-20T09:00:47.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1eacf2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:47.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1eacf2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1e40f6810 0x7fa1e410bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:47.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1eacf2700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e410c1a0 con 0x7fa1e40f5570 2021-11-20T09:00:47.178 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1eacf2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e410c2e0 con 0x7fa1e40f6810 2021-11-20T09:00:47.180 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1e8a8e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:47.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1e8a8e700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44450/0 (socket says 172.21.15.93:44450) 2021-11-20T09:00:47.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1e8a8e700 1 -- 172.21.15.93:0/2459038550 learned_addr learned my addr 172.21.15.93:0/2459038550 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:47.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1e8a8e700 1 -- 172.21.15.93:0/2459038550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1e40f6810 msgr2=0x7fa1e410bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:47.181 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1e3fff700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1e40f6810 0x7fa1e410bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:47.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.166+0000 7fa1e8a8e700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1e40f6810 0x7fa1e410bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:47.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1e8a8e700 1 -- 172.21.15.93:0/2459038550 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8005040 con 0x7fa1e40f5570 2021-11-20T09:00:47.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1e3fff700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1e40f6810 0x7fa1e410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:00:47.182 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1e8a8e700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40604d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fa1d8009f50 tx=0x7fa1d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:47.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1e17fa700 1 -- 172.21.15.93:0/2459038550 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d8018070 con 0x7fa1e40f5570 2021-11-20T09:00:47.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1e17fa700 1 -- 172.21.15.93:0/2459038550 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1d8013410 con 0x7fa1e40f5570 2021-11-20T09:00:47.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1e17fa700 1 -- 172.21.15.93:0/2459038550 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d801c620 con 0x7fa1e40f5570 2021-11-20T09:00:47.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e410c560 con 0x7fa1e40f5570 2021-11-20T09:00:47.183 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.167+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e410cac0 con 0x7fa1e40f5570 2021-11-20T09:00:47.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.168+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e4027a00 con 0x7fa1e40f5570 2021-11-20T09:00:47.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.168+0000 7fa1e17fa700 1 -- 172.21.15.93:0/2459038550 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa1d801d420 con 0x7fa1e40f5570 2021-11-20T09:00:47.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.169+0000 7fa1e17fa700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa1d405c2f0 0x7fa1d405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:47.184 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.169+0000 7fa1e17fa700 1 -- 172.21.15.93:0/2459038550 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa1d80782a0 con 0x7fa1e40f5570 2021-11-20T09:00:47.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.169+0000 7fa1e3fff700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa1d405c2f0 0x7fa1d405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:47.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.169+0000 7fa1e3fff700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa1d405c2f0 0x7fa1d405e7b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa1d000a7e0 tx=0x7fa1d0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:47.185 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.173+0000 7fa1e17fa700 1 -- 172.21.15.93:0/2459038550 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1d804a9f0 con 0x7fa1e40f5570 2021-11-20T09:00:47.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.377+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1e40fdd80 con 0x7fa1e40f5570 2021-11-20T09:00:47.376 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.377+0000 7fa1e17fa700 1 -- 172.21.15.93:0/2459038550 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2189 (secure 0 0 0) 0x7fa1d804e010 con 0x7fa1e40f5570 2021-11-20T09:00:47.377 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:47.377 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:47.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa1d405c2f0 msgr2=0x7fa1d405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:47.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa1d405c2f0 0x7fa1d405e7b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa1d000a7e0 tx=0x7fa1d0008040).stop 2021-11-20T09:00:47.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 msgr2=0x7fa1e40604d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:47.379 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40604d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fa1d8009f50 tx=0x7fa1d8009b30).stop 2021-11-20T09:00:47.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 shutdown_connections 2021-11-20T09:00:47.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa1d405c2f0 0x7fa1d405e7b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:47.380 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1e40f5570 0x7fa1e40604d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:47.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 --2- 172.21.15.93:0/2459038550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1e40f6810 0x7fa1e410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:47.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 >> 172.21.15.93:0/2459038550 conn(0x7fa1e40f0cb0 msgr2=0x7fa1e40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:47.381 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 shutdown_connections 2021-11-20T09:00:47.382 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:47.379+0000 7fa1eacf2700 1 -- 172.21.15.93:0/2459038550 wait complete. 2021-11-20T09:00:47.383 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 84 2021-11-20T09:00:47.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:47.391 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:47.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:47 smithi093 conmon[33522]: cluster 2021-11-20T09:00:46.684850+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:47.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:47 smithi093 conmon[33522]: 835 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:clientreplay 2021-11-20T09:00:47.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:47 smithi093 conmon[33522]: cluster 2021-11-20T09:00:46.684905+0000 mon.smithi093 (mon.0) 836 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:00:47.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:47 smithi093 conmon[33522]: audit 2021-11-20T09:00:47.378142 2021-11-20T09:00:47.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:47 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 837 : audit [DBG] from='client.? 172.21.15.93:0/2459038550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:48.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:47 smithi102 conmon[36472]: cluster 2021-11-20T09:00:46.684850+0000 mon.smithi093 2021-11-20T09:00:48.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:47 smithi102 conmon[36472]: (mon.0) 835 : cluster [DBG] mds.? [v2:172.21.15.102:6826/938076909,v1:172.21.15.102:6827/938076909] up:clientreplay 2021-11-20T09:00:48.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:47 smithi102 conmon[36472]: cluster 2021-11-20T09:00:46.684905+0000 mon.smithi093 (mon.0 2021-11-20T09:00:48.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:47 smithi102 conmon[36472]: ) 836 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:00:48.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:47 smithi102 conmon[36472]: audit 2021-11-20T09:00:48.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:47 smithi102 conmon[36472]: 2021-11-20T09:00:47.378142+0000 mon.smithi093 (mon.0) 837 2021-11-20T09:00:48.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:47 smithi102 conmon[36472]: : audit [DBG] from='client.? 172.21.15.93:0/2459038550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:48.393 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:48.523 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 -- 172.21.15.93:0/3994566197 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 msgr2=0x7fe0700f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:48.524 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 --2- 172.21.15.93:0/3994566197 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0700f7a80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fe05c004660 tx=0x7fe05c009b30).stop 2021-11-20T09:00:48.524 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 -- 172.21.15.93:0/3994566197 shutdown_connections 2021-11-20T09:00:48.524 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 --2- 172.21.15.93:0/3994566197 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0700f7a80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:48.524 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 --2- 172.21.15.93:0/3994566197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0700f4960 0x7fe0700f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:48.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 -- 172.21.15.93:0/3994566197 >> 172.21.15.93:0/3994566197 conn(0x7fe0700f0440 msgr2=0x7fe0700f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:48.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 -- 172.21.15.93:0/3994566197 shutdown_connections 2021-11-20T09:00:48.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.518+0000 7fe076e1c700 1 -- 172.21.15.93:0/3994566197 wait complete. 2021-11-20T09:00:48.525 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe076e1c700 1 Processor -- start 2021-11-20T09:00:48.526 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe076e1c700 1 -- start start 2021-11-20T09:00:48.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe076e1c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0700f4960 0x7fe070106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:48.530 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe076e1c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0701073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:48.530 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe076e1c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070107970 con 0x7fe0700f5300 2021-11-20T09:00:48.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe076e1c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070107ab0 con 0x7fe0700f4960 2021-11-20T09:00:48.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0701073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:48.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0701073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44466/0 (socket says 172.21.15.93:44466) 2021-11-20T09:00:48.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe06ffff700 1 -- 172.21.15.93:0/2676429715 learned_addr learned my addr 172.21.15.93:0/2676429715 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:48.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe074bb8700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0700f4960 0x7fe070106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:48.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe06ffff700 1 -- 172.21.15.93:0/2676429715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0700f4960 msgr2=0x7fe070106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:48.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe06ffff700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0700f4960 0x7fe070106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:48.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe06ffff700 1 -- 172.21.15.93:0/2676429715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c005040 con 0x7fe0700f5300 2021-11-20T09:00:48.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.519+0000 7fe074bb8700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0700f4960 0x7fe070106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:00:48.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.520+0000 7fe06ffff700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0701073c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fe05c009f50 tx=0x7fe05c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:48.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.520+0000 7fe06d7fa700 1 -- 172.21.15.93:0/2676429715 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe05c018070 con 0x7fe0700f5300 2021-11-20T09:00:48.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.520+0000 7fe06d7fa700 1 -- 172.21.15.93:0/2676429715 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe05c013410 con 0x7fe0700f5300 2021-11-20T09:00:48.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.520+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07010c4f0 con 0x7fe0700f5300 2021-11-20T09:00:48.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.520+0000 7fe06d7fa700 1 -- 172.21.15.93:0/2676429715 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe05c01c620 con 0x7fe0700f5300 2021-11-20T09:00:48.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.520+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07010cab0 con 0x7fe0700f5300 2021-11-20T09:00:48.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.521+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070027a00 con 0x7fe0700f5300 2021-11-20T09:00:48.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.521+0000 7fe06d7fa700 1 -- 172.21.15.93:0/2676429715 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe05c01d420 con 0x7fe0700f5300 2021-11-20T09:00:48.535 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.521+0000 7fe06d7fa700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe06005c2a0 0x7fe06005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:48.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.521+0000 7fe06d7fa700 1 -- 172.21.15.93:0/2676429715 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe05c078140 con 0x7fe0700f5300 2021-11-20T09:00:48.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.521+0000 7fe074bb8700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe06005c2a0 0x7fe06005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:48.536 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.522+0000 7fe074bb8700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe06005c2a0 0x7fe06005e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fe07005f7e0 tx=0x7fe064009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:48.537 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.525+0000 7fe06d7fa700 1 -- 172.21.15.93:0/2676429715 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe05c04a9d0 con 0x7fe0700f5300 2021-11-20T09:00:48.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.711+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0700fee50 con 0x7fe0700f5300 2021-11-20T09:00:48.713 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.711+0000 7fe06d7fa700 1 -- 172.21.15.93:0/2676429715 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fe05c04dff0 con 0x7fe0700f5300 2021-11-20T09:00:48.714 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:48.714 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:48.716 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe06005c2a0 msgr2=0x7fe06005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:48.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe06005c2a0 0x7fe06005e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fe07005f7e0 tx=0x7fe064009040).stop 2021-11-20T09:00:48.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 msgr2=0x7fe0701073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:48.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0701073c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fe05c009f50 tx=0x7fe05c009b30).stop 2021-11-20T09:00:48.717 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 shutdown_connections 2021-11-20T09:00:48.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe06005c2a0 0x7fe06005e760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:48.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0700f4960 0x7fe070106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:48.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 --2- 172.21.15.93:0/2676429715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0700f5300 0x7fe0701073c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:48.718 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 >> 172.21.15.93:0/2676429715 conn(0x7fe0700f0440 msgr2=0x7fe0700fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:48.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 shutdown_connections 2021-11-20T09:00:48.719 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:48.713+0000 7fe076e1c700 1 -- 172.21.15.93:0/2676429715 wait complete. 2021-11-20T09:00:48.719 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:48.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:48.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:48 smithi093 conmon[33522]: cluster 2021-11-20T09:00:47.580572+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T09:00:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:48 smithi093 conmon[33522]: 307 : cluster [DBG] pgmap v284: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:48.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:48 smithi093 conmon[33522]: cluster 2021-11-20T09:00:47.694747+0000 mon.smithi093 (mon.0) 838 : cluster [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:standby 2021-11-20T09:00:48.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:48 smithi093 conmon[33522]: cluster 2021-11-20T09:00:48.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:48 smithi093 conmon[33522]: 2021-11-20T09:00:47.694791+0000 mon.smithi093 (mon.0) 839 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:00:49.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:48 smithi102 conmon[36472]: cluster 2021-11-20T09:00:47.580572+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T09:00:49.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:48 smithi102 conmon[36472]: .14162) 307 : cluster [DBG] pgmap v284: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:49.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:48 smithi102 conmon[36472]: cluster 2021-11-20T09:00:47.694747+0000 mon.smithi093 (mon.0) 838 : cluster 2021-11-20T09:00:49.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:48 smithi102 conmon[36472]: [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:standby 2021-11-20T09:00:49.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:48 smithi102 conmon[36472]: cluster 2021-11- 2021-11-20T09:00:49.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:48 smithi102 conmon[36472]: 20T09:00:47.694791+0000 mon.smithi093 (mon.0) 839 : cluster [DBG] 2021-11-20T09:00:49.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:48 smithi102 conmon[36472]: fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:00:49.730 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:49.868 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 -- 172.21.15.93:0/545556276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f6680 msgr2=0x7fa7080f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:49.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 --2- 172.21.15.93:0/545556276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f6680 0x7fa7080f6aa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa704004660 tx=0x7fa704009b30).stop 2021-11-20T09:00:49.869 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 -- 172.21.15.93:0/545556276 shutdown_connections 2021-11-20T09:00:49.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 --2- 172.21.15.93:0/545556276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7080f7920 0x7fa7080f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:49.870 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 --2- 172.21.15.93:0/545556276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f6680 0x7fa7080f6aa0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:49.871 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 -- 172.21.15.93:0/545556276 >> 172.21.15.93:0/545556276 conn(0x7fa7080f1de0 msgr2=0x7fa7080f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:49.871 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 -- 172.21.15.93:0/545556276 shutdown_connections 2021-11-20T09:00:49.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.856+0000 7fa70fa55700 1 -- 172.21.15.93:0/545556276 wait complete. 2021-11-20T09:00:49.872 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70fa55700 1 Processor -- start 2021-11-20T09:00:49.873 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70fa55700 1 -- start start 2021-11-20T09:00:49.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70fa55700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7080f6680 0x7fa708106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:49.874 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70fa55700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f7920 0x7fa708107310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:49.876 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70fa55700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7081078c0 con 0x7fa7080f7920 2021-11-20T09:00:49.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70fa55700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708107a00 con 0x7fa7080f6680 2021-11-20T09:00:49.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70cff0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f7920 0x7fa708107310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:49.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70cff0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f7920 0x7fa708107310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44480/0 (socket says 172.21.15.93:44480) 2021-11-20T09:00:49.878 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70cff0700 1 -- 172.21.15.93:0/314497381 learned_addr learned my addr 172.21.15.93:0/314497381 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:49.878 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70cff0700 1 -- 172.21.15.93:0/314497381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7080f6680 msgr2=0x7fa708106e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:49.879 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70cff0700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7080f6680 0x7fa708106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:49.879 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70cff0700 1 -- 172.21.15.93:0/314497381 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa704005040 con 0x7fa7080f7920 2021-11-20T09:00:49.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70cff0700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f7920 0x7fa708107310 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa6f800e7f0 tx=0x7fa6f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:49.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa6fdffb700 1 -- 172.21.15.93:0/314497381 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6f80092e0 con 0x7fa7080f7920 2021-11-20T09:00:49.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa6fdffb700 1 -- 172.21.15.93:0/314497381 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6f800f040 con 0x7fa7080f7920 2021-11-20T09:00:49.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70810c4a0 con 0x7fa7080f7920 2021-11-20T09:00:49.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.857+0000 7fa6fdffb700 1 -- 172.21.15.93:0/314497381 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6f80146c0 con 0x7fa7080f7920 2021-11-20T09:00:49.882 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.858+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70810ca60 con 0x7fa7080f7920 2021-11-20T09:00:49.882 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.859+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa708027a00 con 0x7fa7080f7920 2021-11-20T09:00:49.883 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.859+0000 7fa6fdffb700 1 -- 172.21.15.93:0/314497381 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa6f801b030 con 0x7fa7080f7920 2021-11-20T09:00:49.883 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.859+0000 7fa6fdffb700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa6f405c2a0 0x7fa6f405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:49.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.859+0000 7fa6fdffb700 1 -- 172.21.15.93:0/314497381 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa6f8076120 con 0x7fa7080f7920 2021-11-20T09:00:49.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.859+0000 7fa70d7f1700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa6f405c2a0 0x7fa6f405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:49.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.860+0000 7fa70d7f1700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa6f405c2a0 0x7fa6f405e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa704013530 tx=0x7fa70400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:49.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:49.863+0000 7fa6fdffb700 1 -- 172.21.15.93:0/314497381 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6f80489e0 con 0x7fa7080f7920 2021-11-20T09:00:49.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:49 smithi093 conmon[33522]: audit 2021-11-20T09:00:48.712554+0000 mon.smithi093 (mon.0) 840 : audit 2021-11-20T09:00:49.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:49 smithi093 conmon[33522]: [DBG] from='client.? 172.21.15.93:0/2676429715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:50.046 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.047+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7080fee10 con 0x7fa7080f7920 2021-11-20T09:00:50.047 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.047+0000 7fa6fdffb700 1 -- 172.21.15.93:0/314497381 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fa6f8019020 con 0x7fa7080f7920 2021-11-20T09:00:50.048 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:50.049 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:50.051 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa6f405c2a0 msgr2=0x7fa6f405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:50.052 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa6f405c2a0 0x7fa6f405e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa704013530 tx=0x7fa70400b040).stop 2021-11-20T09:00:50.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f7920 msgr2=0x7fa708107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:50.056 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f7920 0x7fa708107310 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa6f800e7f0 tx=0x7fa6f800b040).stop 2021-11-20T09:00:50.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 shutdown_connections 2021-11-20T09:00:50.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa6f405c2a0 0x7fa6f405e760 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:50.057 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7080f6680 0x7fa708106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:50.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 --2- 172.21.15.93:0/314497381 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa7080f7920 0x7fa708107310 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:50.058 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 >> 172.21.15.93:0/314497381 conn(0x7fa7080f1de0 msgr2=0x7fa7080fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:50.059 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 shutdown_connections 2021-11-20T09:00:50.059 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:50.049+0000 7fa70fa55700 1 -- 172.21.15.93:0/314497381 wait complete. 2021-11-20T09:00:50.059 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:50.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:50.062 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:50.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:49 smithi102 conmon[36472]: audit 2021-11-20T09:00:48.712554+0000 mon.smithi093 ( 2021-11-20T09:00:50.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:49 smithi102 conmon[36472]: mon.0) 840 : audit [DBG] from='client.? 172.21.15.93:0/2676429715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:50 smithi093 conmon[33522]: cluster 2021-11-20T09:00:49.581127+0000 mgr.smithi093.qdikkm (mgr.14162) 308 : 2021-11-20T09:00:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:50 smithi093 conmon[33522]: cluster [DBG] pgmap v285: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:50.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:50 smithi093 conmon[33522]: audit 2021-11-20T09:00:50.048139+0000 mon.smithi093 (mon.0) 841 : audit [DBG] from='client.? 172.21.15.93:0/314497381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:51.067 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:51.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:50 smithi102 conmon[36472]: cluster 2021-11-20T09:00:49.581127+0000 mgr.smithi093.qdikkm (mgr.14162) 308 : cluster 2021-11-20T09:00:51.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:50 smithi102 conmon[36472]: [DBG] pgmap v285: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:51.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:50 smithi102 conmon[36472]: audit 2021-11-20T09:00:50.048139+0000 mon.smithi093 (mon.0) 841 : audit [DBG] from='client.? 172.21.15.93:0/314497381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:51.206 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- 172.21.15.93:0/3836375608 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 msgr2=0x7f87480f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:51.207 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 --2- 172.21.15.93:0/3836375608 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5dd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f8734004660 tx=0x7f8734009b30).stop 2021-11-20T09:00:51.207 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- 172.21.15.93:0/3836375608 shutdown_connections 2021-11-20T09:00:51.208 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 --2- 172.21.15.93:0/3836375608 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5dd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:51.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 --2- 172.21.15.93:0/3836375608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87480f6c40 0x7f87480f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:51.209 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- 172.21.15.93:0/3836375608 >> 172.21.15.93:0/3836375608 conn(0x7f87480f04a0 msgr2=0x7f87480f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:51.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- 172.21.15.93:0/3836375608 shutdown_connections 2021-11-20T09:00:51.210 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- 172.21.15.93:0/3836375608 wait complete. 2021-11-20T09:00:51.212 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 Processor -- start 2021-11-20T09:00:51.216 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- start start 2021-11-20T09:00:51.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:51.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87480f6c40 0x7f874810e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:51.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87481079f0 con 0x7f87480f5950 2021-11-20T09:00:51.217 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.194+0000 7f874eaac700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874810e500 con 0x7f87480f6c40 2021-11-20T09:00:51.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874c848700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:51.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874c848700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44486/0 (socket says 172.21.15.93:44486) 2021-11-20T09:00:51.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874c848700 1 -- 172.21.15.93:0/2916459921 learned_addr learned my addr 172.21.15.93:0/2916459921 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:51.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f8747fff700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87480f6c40 0x7f874810e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:51.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874c848700 1 -- 172.21.15.93:0/2916459921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87480f6c40 msgr2=0x7f874810e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:51.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874c848700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87480f6c40 0x7f874810e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:51.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874c848700 1 -- 172.21.15.93:0/2916459921 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8734005040 con 0x7f87480f5950 2021-11-20T09:00:51.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874c848700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5350 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f873c0026d0 tx=0x7f873c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:51.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f87457fa700 1 -- 172.21.15.93:0/2916459921 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f873c009320 con 0x7f87480f5950 2021-11-20T09:00:51.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f87457fa700 1 -- 172.21.15.93:0/2916459921 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f873c00f040 con 0x7f87480f5950 2021-11-20T09:00:51.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874810e700 con 0x7f87480f5950 2021-11-20T09:00:51.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f87457fa700 1 -- 172.21.15.93:0/2916459921 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f873c0147e0 con 0x7f87480f5950 2021-11-20T09:00:51.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.195+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874810ec90 con 0x7f87480f5950 2021-11-20T09:00:51.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.196+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8748027a00 con 0x7f87480f5950 2021-11-20T09:00:51.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.196+0000 7f87457fa700 1 -- 172.21.15.93:0/2916459921 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f873c009480 con 0x7f87480f5950 2021-11-20T09:00:51.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.197+0000 7f87457fa700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f873805c2a0 0x7f873805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:51.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.197+0000 7f87457fa700 1 -- 172.21.15.93:0/2916459921 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f873c076a60 con 0x7f87480f5950 2021-11-20T09:00:51.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.197+0000 7f8747fff700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f873805c2a0 0x7f873805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:51.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.197+0000 7f8747fff700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f873805c2a0 0x7f873805e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8734004e30 tx=0x7f8734009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:51.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.201+0000 7f87457fa700 1 -- 172.21.15.93:0/2916459921 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f873c049270 con 0x7f87480f5950 2021-11-20T09:00:51.400 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.401+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87480fddf0 con 0x7f87480f5950 2021-11-20T09:00:51.404 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.401+0000 7f87457fa700 1 -- 172.21.15.93:0/2916459921 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f873c04c890 con 0x7f87480f5950 2021-11-20T09:00:51.405 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:51.406 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:51.408 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f873805c2a0 msgr2=0x7f873805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:51.408 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f873805c2a0 0x7f873805e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8734004e30 tx=0x7f8734009b30).stop 2021-11-20T09:00:51.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 msgr2=0x7f87480f5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:51.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5350 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f873c0026d0 tx=0x7f873c009de0).stop 2021-11-20T09:00:51.409 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 shutdown_connections 2021-11-20T09:00:51.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f873805c2a0 0x7f873805e760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:51.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f87480f5950 0x7f87480f5350 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:51.410 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 --2- 172.21.15.93:0/2916459921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87480f6c40 0x7f874810e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:51.411 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 >> 172.21.15.93:0/2916459921 conn(0x7f87480f04a0 msgr2=0x7f87480fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:51.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 shutdown_connections 2021-11-20T09:00:51.412 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:51.403+0000 7f874eaac700 1 -- 172.21.15.93:0/2916459921 wait complete. 2021-11-20T09:00:51.412 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:51.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:51.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:51.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:51 smithi093 conmon[33522]: audit 2021-11-20T09:00:51.402104+0000 mon.smithi093 (mon.0) 842 : audit [DBG] from='client.? 172.21.15.93:0/2916459921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:52.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:51 smithi102 conmon[36472]: audit 2021-11-20T09:00:51 2021-11-20T09:00:52.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:51 smithi102 conmon[36472]: .402104+0000 mon.smithi093 (mon.0) 842 : audit [DBG] from='client.? 172.21.15.93:0/2916459921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:52.415 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:52.557 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- 172.21.15.93:0/3019969327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 msgr2=0x7f860c0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:52.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 --2- 172.21.15.93:0/3019969327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c0f58f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f8608004660 tx=0x7f8608009b30).stop 2021-11-20T09:00:52.558 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- 172.21.15.93:0/3019969327 shutdown_connections 2021-11-20T09:00:52.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 --2- 172.21.15.93:0/3019969327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f860c0f6770 0x7f860c0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:52.559 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 --2- 172.21.15.93:0/3019969327 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c0f58f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:52.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- 172.21.15.93:0/3019969327 >> 172.21.15.93:0/3019969327 conn(0x7f860c0f0c30 msgr2=0x7f860c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:52.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- 172.21.15.93:0/3019969327 shutdown_connections 2021-11-20T09:00:52.560 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- 172.21.15.93:0/3019969327 wait complete. 2021-11-20T09:00:52.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 Processor -- start 2021-11-20T09:00:52.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- start start 2021-11-20T09:00:52.561 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:52.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f860c0f6770 0x7f860c10bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:52.562 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c060540 con 0x7f860c0f54d0 2021-11-20T09:00:52.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.551+0000 7f8613533700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c060680 con 0x7f860c0f6770 2021-11-20T09:00:52.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8612531700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:52.563 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8612531700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c109a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44498/0 (socket says 172.21.15.93:44498) 2021-11-20T09:00:52.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8612531700 1 -- 172.21.15.93:0/552477757 learned_addr learned my addr 172.21.15.93:0/552477757 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:52.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8612531700 1 -- 172.21.15.93:0/552477757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f860c0f6770 msgr2=0x7f860c10bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:52.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8612531700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f860c0f6770 0x7f860c10bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:52.564 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8612531700 1 -- 172.21.15.93:0/552477757 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608005040 con 0x7f860c0f54d0 2021-11-20T09:00:52.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8612531700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c109a80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f8608004e30 tx=0x7f8608009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:52.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8602ffd700 1 -- 172.21.15.93:0/552477757 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8608018070 con 0x7f860c0f54d0 2021-11-20T09:00:52.565 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c10c4d0 con 0x7f860c0f54d0 2021-11-20T09:00:52.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8602ffd700 1 -- 172.21.15.93:0/552477757 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8608013410 con 0x7f860c0f54d0 2021-11-20T09:00:52.566 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8602ffd700 1 -- 172.21.15.93:0/552477757 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f860801c3c0 con 0x7f860c0f54d0 2021-11-20T09:00:52.567 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.552+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c10ca30 con 0x7f860c0f54d0 2021-11-20T09:00:52.567 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.553+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c045ad0 con 0x7f860c0f54d0 2021-11-20T09:00:52.568 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.554+0000 7f8602ffd700 1 -- 172.21.15.93:0/552477757 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f860801d420 con 0x7f860c0f54d0 2021-11-20T09:00:52.568 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.554+0000 7f8602ffd700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f85f805c300 0x7f85f805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:52.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.554+0000 7f8602ffd700 1 -- 172.21.15.93:0/552477757 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8608078010 con 0x7f860c0f54d0 2021-11-20T09:00:52.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.554+0000 7f8611d30700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f85f805c300 0x7f85f805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:52.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.554+0000 7f8611d30700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f85f805c300 0x7f85f805e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f85fc004a10 tx=0x7f85fc0099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:52.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.558+0000 7f8602ffd700 1 -- 172.21.15.93:0/552477757 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f860804a820 con 0x7f860c0f54d0 2021-11-20T09:00:52.743 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.743+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f860c0fdd70 con 0x7f860c0f54d0 2021-11-20T09:00:52.744 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.744+0000 7f8602ffd700 1 -- 172.21.15.93:0/552477757 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f860804de40 con 0x7f860c0f54d0 2021-11-20T09:00:52.745 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:52.745 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:52.747 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f85f805c300 msgr2=0x7f85f805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:52.748 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f85f805c300 0x7f85f805e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f85fc004a10 tx=0x7f85fc0099e0).stop 2021-11-20T09:00:52.748 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 msgr2=0x7f860c109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:52.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c109a80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f8608004e30 tx=0x7f8608009b30).stop 2021-11-20T09:00:52.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 shutdown_connections 2021-11-20T09:00:52.749 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f85f805c300 0x7f85f805e7c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:52.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f860c0f54d0 0x7f860c109a80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:52.750 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 --2- 172.21.15.93:0/552477757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f860c0f6770 0x7f860c10bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:52.751 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 >> 172.21.15.93:0/552477757 conn(0x7f860c0f0c30 msgr2=0x7f860c0fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:52.751 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 shutdown_connections 2021-11-20T09:00:52.752 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:52.745+0000 7f8613533700 1 -- 172.21.15.93:0/552477757 wait complete. 2021-11-20T09:00:52.752 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:52.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:52.756 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:52.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:52 smithi093 conmon[33522]: cluster 2021-11-20T09:00:51.581592+0000 mgr.smithi093.qdikkm (mgr.14162) 309 : cluster [DBG] 2021-11-20T09:00:52.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:52 smithi093 conmon[33522]: pgmap v286: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:53.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:52 smithi102 conmon[36472]: cluster 2021-11-20T09:00:51.581592+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T09:00:53.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:52 smithi102 conmon[36472]: .14162) 309 : cluster [DBG] pgmap v286: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:00:53.758 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:53.882 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 -- 172.21.15.93:0/144953356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640600c0 msgr2=0x7f80640f9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:53.883 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 --2- 172.21.15.93:0/144953356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640600c0 0x7f80640f9300 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f8054004660 tx=0x7f8054009b30).stop 2021-11-20T09:00:53.883 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 -- 172.21.15.93:0/144953356 shutdown_connections 2021-11-20T09:00:53.883 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 --2- 172.21.15.93:0/144953356 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80640f9880 0x7f80640fbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:53.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 --2- 172.21.15.93:0/144953356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640600c0 0x7f80640f9300 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:53.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 -- 172.21.15.93:0/144953356 >> 172.21.15.93:0/144953356 conn(0x7f80640f0ce0 msgr2=0x7f80640f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:53.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 -- 172.21.15.93:0/144953356 shutdown_connections 2021-11-20T09:00:53.884 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.878+0000 7f806ac54700 1 -- 172.21.15.93:0/144953356 wait complete. 2021-11-20T09:00:53.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f806ac54700 1 Processor -- start 2021-11-20T09:00:53.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f806ac54700 1 -- start start 2021-11-20T09:00:53.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f806ac54700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80640600c0 0x7f8064106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:53.885 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f806ac54700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640f9880 0x7f80641073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:53.886 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f806ac54700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8064107950 con 0x7f80640f9880 2021-11-20T09:00:53.886 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f806ac54700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8064107a90 con 0x7f80640600c0 2021-11-20T09:00:53.886 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f8063fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640f9880 0x7f80641073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:53.886 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f8063fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640f9880 0x7f80641073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44510/0 (socket says 172.21.15.93:44510) 2021-11-20T09:00:53.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f8063fff700 1 -- 172.21.15.93:0/1567661817 learned_addr learned my addr 172.21.15.93:0/1567661817 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:53.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f80689f0700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80640600c0 0x7f8064106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:53.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f8063fff700 1 -- 172.21.15.93:0/1567661817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80640600c0 msgr2=0x7f8064106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:53.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f8063fff700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80640600c0 0x7f8064106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:53.887 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f8063fff700 1 -- 172.21.15.93:0/1567661817 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054005040 con 0x7f80640f9880 2021-11-20T09:00:53.888 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f80689f0700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80640600c0 0x7f8064106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:00:53.888 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.879+0000 7f8063fff700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640f9880 0x7f80641073a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f804c00d7f0 tx=0x7f804c004e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:53.888 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.880+0000 7f80617fa700 1 -- 172.21.15.93:0/1567661817 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f804c00b4f0 con 0x7f80640f9880 2021-11-20T09:00:53.888 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.880+0000 7f80617fa700 1 -- 172.21.15.93:0/1567661817 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f804c00f040 con 0x7f80640f9880 2021-11-20T09:00:53.889 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.880+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806410c530 con 0x7f80640f9880 2021-11-20T09:00:53.889 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.880+0000 7f80617fa700 1 -- 172.21.15.93:0/1567661817 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f804c0139b0 con 0x7f80640f9880 2021-11-20T09:00:53.890 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.880+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806410caf0 con 0x7f80640f9880 2021-11-20T09:00:53.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.881+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8064027a00 con 0x7f80640f9880 2021-11-20T09:00:53.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.881+0000 7f80617fa700 1 -- 172.21.15.93:0/1567661817 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f804c01a030 con 0x7f80640f9880 2021-11-20T09:00:53.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.881+0000 7f80617fa700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f805005c2a0 0x7f805005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:53.891 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.881+0000 7f80617fa700 1 -- 172.21.15.93:0/1567661817 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f804c076120 con 0x7f80640f9880 2021-11-20T09:00:53.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.881+0000 7f80689f0700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f805005c2a0 0x7f805005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:53.892 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.882+0000 7f80689f0700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f805005c2a0 0x7f805005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8054005010 tx=0x7f8054009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:53.893 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:53.885+0000 7f80617fa700 1 -- 172.21.15.93:0/1567661817 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f804c0489e0 con 0x7f80640f9880 2021-11-20T09:00:54.070 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.068+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80640fee50 con 0x7f80640f9880 2021-11-20T09:00:54.070 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.068+0000 7f80617fa700 1 -- 172.21.15.93:0/1567661817 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f804c04c000 con 0x7f80640f9880 2021-11-20T09:00:54.071 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:54.071 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:54.072 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f805005c2a0 msgr2=0x7f805005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:54.072 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f805005c2a0 0x7f805005e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8054005010 tx=0x7f8054009b30).stop 2021-11-20T09:00:54.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640f9880 msgr2=0x7f80641073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:54.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640f9880 0x7f80641073a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f804c00d7f0 tx=0x7f804c004e60).stop 2021-11-20T09:00:54.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 shutdown_connections 2021-11-20T09:00:54.073 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f805005c2a0 0x7f805005e760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:54.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80640600c0 0x7f8064106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:54.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 --2- 172.21.15.93:0/1567661817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80640f9880 0x7f80641073a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:54.074 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 >> 172.21.15.93:0/1567661817 conn(0x7f80640f0ce0 msgr2=0x7f80640f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:54.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 shutdown_connections 2021-11-20T09:00:54.075 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:54.070+0000 7f806ac54700 1 -- 172.21.15.93:0/1567661817 wait complete. 2021-11-20T09:00:54.075 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:54.079 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:53 smithi093 conmon[33522]: audit 2021-11-20T09:00:52.744734+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:54.080 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:53 smithi093 conmon[33522]: 843 : audit [DBG] from='client.? 172.21.15.93:0/552477757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:54.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:54.082 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:54.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:53 smithi102 conmon[36472]: audit 2021-11-20T09:00:52.744734+0000 mon.smithi093 (mon. 2021-11-20T09:00:54.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:53 smithi102 conmon[36472]: 0) 843 : audit [DBG] from='client.? 172.21.15.93:0/552477757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:55.084 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:55.113 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:54 smithi093 conmon[33522]: cluster 2021-11-20T09:00:53.581969+0000 mgr.smithi093.qdikkm (mgr.14162) 310 : cluster [DBG] 2021-11-20T09:00:55.114 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:54 smithi093 conmon[33522]: pgmap v287: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 196 B/s wr, 1 op/s 2021-11-20T09:00:55.114 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:54 smithi093 conmon[33522]: audit 2021-11-20T09:00:54.069446+0000 mon.smithi093 (mon.0) 844 : audit [DBG] from='client.? 172.21.15.93:0/1567661817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:55.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:54 smithi102 conmon[36472]: cluster 2021-11-20T09:00:53.581969+0000 2021-11-20T09:00:55.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:54 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 310 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 196 B/s wr, 1 op/s 2021-11-20T09:00:55.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:54 smithi102 conmon[36472]: audit 2021-11-20T09:00:54. 2021-11-20T09:00:55.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:54 smithi102 conmon[36472]: 069446+0000 mon.smithi093 (mon.0) 844 : audit [DBG] from='client.? 172.21.15.93:0/1567661817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:55.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 -- 172.21.15.93:0/2548251772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f7920 msgr2=0x7f53300f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:55.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 --2- 172.21.15.93:0/2548251772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f7920 0x7f53300f7d80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f5324004660 tx=0x7f5324009b30).stop 2021-11-20T09:00:55.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 -- 172.21.15.93:0/2548251772 shutdown_connections 2021-11-20T09:00:55.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 --2- 172.21.15.93:0/2548251772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f7920 0x7f53300f7d80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:55.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 --2- 172.21.15.93:0/2548251772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53300f6680 0x7f53300f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:55.225 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 -- 172.21.15.93:0/2548251772 >> 172.21.15.93:0/2548251772 conn(0x7f53300f1de0 msgr2=0x7f53300f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:55.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 -- 172.21.15.93:0/2548251772 shutdown_connections 2021-11-20T09:00:55.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.220+0000 7f5335880700 1 -- 172.21.15.93:0/2548251772 wait complete. 2021-11-20T09:00:55.226 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f5335880700 1 Processor -- start 2021-11-20T09:00:55.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f5335880700 1 -- start start 2021-11-20T09:00:55.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f5335880700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f6680 0x7f5330106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:55.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f5335880700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53300f7920 0x7f5330107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:55.227 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f5335880700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330107930 con 0x7f53300f6680 2021-11-20T09:00:55.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f5335880700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330107a70 con 0x7f53300f7920 2021-11-20T09:00:55.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f532effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f6680 0x7f5330106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:55.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f532effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f6680 0x7f5330106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44518/0 (socket says 172.21.15.93:44518) 2021-11-20T09:00:55.228 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.221+0000 7f532effd700 1 -- 172.21.15.93:0/1942325536 learned_addr learned my addr 172.21.15.93:0/1942325536 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:55.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f532e7fc700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53300f7920 0x7f5330107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:55.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f532effd700 1 -- 172.21.15.93:0/1942325536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53300f7920 msgr2=0x7f5330107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:55.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f532effd700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53300f7920 0x7f5330107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:55.229 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f532effd700 1 -- 172.21.15.93:0/1942325536 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5324005040 con 0x7f53300f6680 2021-11-20T09:00:55.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f532effd700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f6680 0x7f5330106e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f5320002700 tx=0x7f5320009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:55.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f53177fe700 1 -- 172.21.15.93:0/1942325536 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5320009320 con 0x7f53300f6680 2021-11-20T09:00:55.230 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f53177fe700 1 -- 172.21.15.93:0/1942325536 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f532000f040 con 0x7f53300f6680 2021-11-20T09:00:55.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f53177fe700 1 -- 172.21.15.93:0/1942325536 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5320014900 con 0x7f53300f6680 2021-11-20T09:00:55.232 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533010c510 con 0x7f53300f6680 2021-11-20T09:00:55.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.222+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533010ca80 con 0x7f53300f6680 2021-11-20T09:00:55.233 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.223+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5330027a00 con 0x7f53300f6680 2021-11-20T09:00:55.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.223+0000 7f53177fe700 1 -- 172.21.15.93:0/1942325536 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5320009480 con 0x7f53300f6680 2021-11-20T09:00:55.234 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.224+0000 7f53177fe700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f531805c2a0 0x7f531805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:55.235 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.224+0000 7f53177fe700 1 -- 172.21.15.93:0/1942325536 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f53200766d0 con 0x7f53300f6680 2021-11-20T09:00:55.239 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.224+0000 7f532e7fc700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f531805c2a0 0x7f531805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:55.239 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.224+0000 7f532e7fc700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f531805c2a0 0x7f531805e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f5324005010 tx=0x7f5324009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:55.240 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.228+0000 7f53177fe700 1 -- 172.21.15.93:0/1942325536 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5320048ee0 con 0x7f53300f6680 2021-11-20T09:00:55.417 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.418+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53300fee10 con 0x7f53300f6680 2021-11-20T09:00:55.417 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.418+0000 7f53177fe700 1 -- 172.21.15.93:0/1942325536 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f532004c500 con 0x7f53300f6680 2021-11-20T09:00:55.417 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:55.418 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:55.419 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f531805c2a0 msgr2=0x7f531805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:55.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f531805c2a0 0x7f531805e760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f5324005010 tx=0x7f5324009b30).stop 2021-11-20T09:00:55.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f6680 msgr2=0x7f5330106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:55.420 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f6680 0x7f5330106e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f5320002700 tx=0x7f5320009d80).stop 2021-11-20T09:00:55.421 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 shutdown_connections 2021-11-20T09:00:55.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f531805c2a0 0x7f531805e760 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:55.422 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53300f6680 0x7f5330106e90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:55.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 --2- 172.21.15.93:0/1942325536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53300f7920 0x7f5330107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:55.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 >> 172.21.15.93:0/1942325536 conn(0x7f53300f1de0 msgr2=0x7f53300fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:55.423 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 shutdown_connections 2021-11-20T09:00:55.424 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:55.420+0000 7f5335880700 1 -- 172.21.15.93:0/1942325536 wait complete. 2021-11-20T09:00:55.424 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:55.433 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:55.433 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:56.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:55 smithi102 conmon[36472]: audit 2021-11-20T09:00:55.419168+0000 mon.smithi093 (mon.0) 2021-11-20T09:00:56.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:55 smithi102 conmon[36472]: 845 : audit [DBG] from='client.? 172.21.15.93:0/1942325536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:56.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:55 smithi093 conmon[33522]: audit 2021-11-20T09:00:55.419168+0000 2021-11-20T09:00:56.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:55 smithi093 conmon[33522]: mon.smithi093 (mon.0) 845 : audit [DBG] from='client.? 172.21.15.93:0/1942325536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:56.434 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:56.568 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/1599825076 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f7d90 msgr2=0x7fc7d80f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:56.568 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/1599825076 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f7d90 0x7fc7d80f81b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fc7c80099e0 tx=0x7fc7c8004d10).stop 2021-11-20T09:00:56.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/1599825076 shutdown_connections 2021-11-20T09:00:56.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/1599825076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d80f4780 0x7fc7d80f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:00:56.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/1599825076 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f7d90 0x7fc7d80f81b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:56.569 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/1599825076 >> 172.21.15.93:0/1599825076 conn(0x7fc7d80f04a0 msgr2=0x7fc7d80f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:56.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/1599825076 shutdown_connections 2021-11-20T09:00:56.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.567+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/1599825076 wait complete. 2021-11-20T09:00:56.570 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7dd8e7700 1 Processor -- start 2021-11-20T09:00:56.571 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7dd8e7700 1 -- start start 2021-11-20T09:00:56.571 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7dd8e7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f4780 0x7fc7d8107130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:56.571 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7dd8e7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d80f7d90 0x7fc7d8107620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:56.571 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7dd8e7700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d8107c60 con 0x7fc7d80f4780 2021-11-20T09:00:56.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7dd8e7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d8107da0 con 0x7fc7d80f7d90 2021-11-20T09:00:56.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7d6ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f4780 0x7fc7d8107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:56.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7d6ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f4780 0x7fc7d8107130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44528/0 (socket says 172.21.15.93:44528) 2021-11-20T09:00:56.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7d6ffd700 1 -- 172.21.15.93:0/3640016957 learned_addr learned my addr 172.21.15.93:0/3640016957 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:56.572 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.568+0000 7fc7d67fc700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d80f7d90 0x7fc7d8107620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:56.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7d6ffd700 1 -- 172.21.15.93:0/3640016957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d80f7d90 msgr2=0x7fc7d8107620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:56.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7d6ffd700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d80f7d90 0x7fc7d8107620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:56.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7d6ffd700 1 -- 172.21.15.93:0/3640016957 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8009640 con 0x7fc7d80f4780 2021-11-20T09:00:56.573 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7d6ffd700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f4780 0x7fc7d8107130 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc7c8007830 tx=0x7fc7c8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:56.574 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7bf7fe700 1 -- 172.21.15.93:0/3640016957 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c800e030 con 0x7fc7d80f4780 2021-11-20T09:00:56.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7bf7fe700 1 -- 172.21.15.93:0/3640016957 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7c801d8d0 con 0x7fc7d80f4780 2021-11-20T09:00:56.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d810c7e0 con 0x7fc7d80f4780 2021-11-20T09:00:56.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7bf7fe700 1 -- 172.21.15.93:0/3640016957 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c8018070 con 0x7fc7d80f4780 2021-11-20T09:00:56.575 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.569+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d810ccc0 con 0x7fc7d80f4780 2021-11-20T09:00:56.577 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.570+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d8027a00 con 0x7fc7d80f4780 2021-11-20T09:00:56.577 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.570+0000 7fc7bf7fe700 1 -- 172.21.15.93:0/3640016957 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc7c801da40 con 0x7fc7d80f4780 2021-11-20T09:00:56.577 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.570+0000 7fc7bf7fe700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc7c005c290 0x7fc7c005e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:56.578 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.570+0000 7fc7bf7fe700 1 -- 172.21.15.93:0/3640016957 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc7c8078290 con 0x7fc7d80f4780 2021-11-20T09:00:56.578 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.571+0000 7fc7d67fc700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc7c005c290 0x7fc7c005e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:56.578 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.571+0000 7fc7d67fc700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc7c005c290 0x7fc7c005e750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fc7d80f7b60 tx=0x7fc7cc008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:56.579 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.574+0000 7fc7bf7fe700 1 -- 172.21.15.93:0/3640016957 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7c804aaa0 con 0x7fc7d80f4780 2021-11-20T09:00:56.765 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.764+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7d80fefe0 con 0x7fc7d80f4780 2021-11-20T09:00:56.766 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.764+0000 7fc7bf7fe700 1 -- 172.21.15.93:0/3640016957 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fc7c804e0c0 con 0x7fc7d80f4780 2021-11-20T09:00:56.767 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:56.768 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:56.774 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc7c005c290 msgr2=0x7fc7c005e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:56.774 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc7c005c290 0x7fc7c005e750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fc7d80f7b60 tx=0x7fc7cc008040).stop 2021-11-20T09:00:56.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f4780 msgr2=0x7fc7d8107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:56.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f4780 0x7fc7d8107130 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc7c8007830 tx=0x7fc7c8004d10).stop 2021-11-20T09:00:56.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 shutdown_connections 2021-11-20T09:00:56.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc7c005c290 0x7fc7c005e750 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:56.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7d80f4780 0x7fc7d8107130 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:56.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 --2- 172.21.15.93:0/3640016957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d80f7d90 0x7fc7d8107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:56.777 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 >> 172.21.15.93:0/3640016957 conn(0x7fc7d80f04a0 msgr2=0x7fc7d80fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:56.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 shutdown_connections 2021-11-20T09:00:56.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:56.766+0000 7fc7dd8e7700 1 -- 172.21.15.93:0/3640016957 wait complete. 2021-11-20T09:00:56.779 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:56.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:56.782 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:57.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:56 smithi102 conmon[36472]: cluster 2021-11-20T09:00:55.582344+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T09:00:57.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:56 smithi102 conmon[36472]: ) 311 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:57.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:56 smithi102 conmon[36472]: audit 2021-11-20T09:00:56.625541+0000 mon.smithi093 (mon.0) 846 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T09:00:57.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:56 smithi102 conmon[36472]: audit 2021-11-20T09:00:56.626921+0000 2021-11-20T09:00:57.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:56 smithi102 conmon[36472]: mon.smithi093 (mon.0) 847 : audit [INF] 2021-11-20T09:00:57.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:56 smithi102 conmon[36472]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T09:00:57.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:56 smithi093 conmon[33522]: cluster 2021-11-20T09:00:55. 2021-11-20T09:00:57.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:56 smithi093 conmon[33522]: 582344+0000 mgr.smithi093.qdikkm (mgr.14162) 311 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:57.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:56 smithi093 conmon[33522]: audit 2021-11-20T09:00:56.625541+0000 mon.smithi093 (mon 2021-11-20T09:00:57.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:56 smithi093 conmon[33522]: .0) 846 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/mirror_snapshot_schedule"}]: dispatch 2021-11-20T09:00:57.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:56 smithi093 conmon[33522]: audit 2021-11-20 2021-11-20T09:00:57.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:56 smithi093 conmon[33522]: T09:00:56.626921+0000 mon.smithi093 (mon. 2021-11-20T09:00:57.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:56 smithi093 conmon[33522]: 0) 847 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi093.qdikkm/trash_purge_schedule"}]: dispatch 2021-11-20T09:00:57.791 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:57.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 -- 172.21.15.93:0/4277205938 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f4d20 msgr2=0x7f2a0c0f5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:57.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 --2- 172.21.15.93:0/4277205938 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f4d20 0x7f2a0c0f5140 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f29fc004660 tx=0x7f29fc009b30).stop 2021-11-20T09:00:57.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 -- 172.21.15.93:0/4277205938 shutdown_connections 2021-11-20T09:00:57.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 --2- 172.21.15.93:0/4277205938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a0c0f5fc0 0x7f2a0c0f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:57.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 --2- 172.21.15.93:0/4277205938 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f4d20 0x7f2a0c0f5140 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:57.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 -- 172.21.15.93:0/4277205938 >> 172.21.15.93:0/4277205938 conn(0x7f2a0c0f0440 msgr2=0x7f2a0c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:57.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 -- 172.21.15.93:0/4277205938 shutdown_connections 2021-11-20T09:00:57.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.917+0000 7f2a12468700 1 -- 172.21.15.93:0/4277205938 wait complete. 2021-11-20T09:00:57.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a12468700 1 Processor -- start 2021-11-20T09:00:57.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a12468700 1 -- start start 2021-11-20T09:00:57.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a12468700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a0c0f4d20 0x7f2a0c0f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:57.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a12468700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f5fc0 0x7f2a0c10be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:57.936 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a12468700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c10c300 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.936 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a12468700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c10c440 con 0x7f2a0c0f4d20 2021-11-20T09:00:57.936 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f5fc0 0x7f2a0c10be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:57.938 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f5fc0 0x7f2a0c10be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44538/0 (socket says 172.21.15.93:44538) 2021-11-20T09:00:57.938 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0b7fe700 1 -- 172.21.15.93:0/3457842409 learned_addr learned my addr 172.21.15.93:0/3457842409 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:57.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0bfff700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a0c0f4d20 0x7f2a0c0f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:57.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0b7fe700 1 -- 172.21.15.93:0/3457842409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a0c0f4d20 msgr2=0x7f2a0c0f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:57.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0b7fe700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a0c0f4d20 0x7f2a0c0f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:57.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0b7fe700 1 -- 172.21.15.93:0/3457842409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc005040 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.918+0000 7f2a0b7fe700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f5fc0 0x7f2a0c10be10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f2a0000c560 tx=0x7f2a00007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:57.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.919+0000 7f2a08ff9700 1 -- 172.21.15.93:0/3457842409 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a0000f650 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.919+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c10c6a0 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.919+0000 7f2a08ff9700 1 -- 172.21.15.93:0/3457842409 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a0000fc60 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.919+0000 7f2a08ff9700 1 -- 172.21.15.93:0/3457842409 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a00012c10 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.919+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c10cbe0 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.920+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c0fd4b0 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.920+0000 7f2a08ff9700 1 -- 172.21.15.93:0/3457842409 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2a00010070 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.920+0000 7f2a08ff9700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29f405c2f0 0x7f29f405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:57.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.920+0000 7f2a08ff9700 1 -- 172.21.15.93:0/3457842409 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2a000760e0 con 0x7f2a0c0f5fc0 2021-11-20T09:00:57.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.920+0000 7f2a0bfff700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29f405c2f0 0x7f29f405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:57.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.921+0000 7f2a0bfff700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29f405c2f0 0x7f29f405e7b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f29fc004630 tx=0x7f29fc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:57.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:57.924+0000 7f2a08ff9700 1 -- 172.21.15.93:0/3457842409 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a000452b0 con 0x7f2a0c0f5fc0 2021-11-20T09:00:58.108 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.109+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a0c10d4b0 con 0x7f2a0c0f5fc0 2021-11-20T09:00:58.109 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.110+0000 7f2a08ff9700 1 -- 172.21.15.93:0/3457842409 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f2a0004c260 con 0x7f2a0c0f5fc0 2021-11-20T09:00:58.111 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:58.113 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29f405c2f0 msgr2=0x7f29f405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:58.113 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29f405c2f0 0x7f29f405e7b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f29fc004630 tx=0x7f29fc009b30).stop 2021-11-20T09:00:58.113 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f5fc0 msgr2=0x7f2a0c10be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:58.115 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f5fc0 0x7f2a0c10be10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f2a0000c560 tx=0x7f2a00007e20).stop 2021-11-20T09:00:58.115 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 shutdown_connections 2021-11-20T09:00:58.115 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f29f405c2f0 0x7f29f405e7b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:58.115 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a0c0f4d20 0x7f2a0c0f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:58.116 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 --2- 172.21.15.93:0/3457842409 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a0c0f5fc0 0x7f2a0c10be10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:58.116 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 >> 172.21.15.93:0/3457842409 conn(0x7f2a0c0f0440 msgr2=0x7f2a0c0fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:58.116 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 shutdown_connections 2021-11-20T09:00:58.116 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:58.111+0000 7f2a12468700 1 -- 172.21.15.93:0/3457842409 wait complete. 2021-11-20T09:00:58.117 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:58.121 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:57 smithi093 conmon[33522]: audit 2021-11-20T09:00:56.765134+0000 mon.smithi093 (mon.0) 848 : 2021-11-20T09:00:58.121 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:57 smithi093 conmon[33522]: audit [DBG] from='client.? 172.21.15.93:0/3640016957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:58.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:58.124 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:00:58.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:57 smithi102 conmon[36472]: audit 2021-11-20T09:00:56.765134+0000 mon.smithi093 (mon.0 2021-11-20T09:00:58.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:57 smithi102 conmon[36472]: ) 848 : audit [DBG] from='client.? 172.21.15.93:0/3640016957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:59.124 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:00:59.153 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:58 smithi093 conmon[33522]: cluster 2021-11-20T09:00:57.582992+0000 mgr.smithi093.qdikkm (mgr.14162) 312 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:59.154 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:58 smithi093 conmon[33522]: audit 2021-11-20T09:00:58.110732+0000 mon.smithi093 (mon.0) 849 : audit [DBG] from='client.? 172.21.15.93:0/3457842409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:59.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:58 smithi102 conmon[36472]: cluster 2021-11-20T09:00:57.582992+0000 mgr.smithi093.qdikkm (mgr.14162) 312 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:00:59.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:58 smithi102 conmon[36472]: audit 2021-11-20T09:00:58.110732+0000 mon.smithi093 (mon.0) 849 : audit [DBG] from='client.? 172.21.15.93:0/3457842409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:00:59.253 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 -- 172.21.15.93:0/120326810 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 msgr2=0x7fe94c0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:59.254 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 --2- 172.21.15.93:0/120326810 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c0f7d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fe940004660 tx=0x7fe940009b30).stop 2021-11-20T09:00:59.254 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 -- 172.21.15.93:0/120326810 shutdown_connections 2021-11-20T09:00:59.254 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 --2- 172.21.15.93:0/120326810 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c0f7d80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:59.254 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 --2- 172.21.15.93:0/120326810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe94c0f6680 0x7fe94c0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:59.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 -- 172.21.15.93:0/120326810 >> 172.21.15.93:0/120326810 conn(0x7fe94c0f1de0 msgr2=0x7fe94c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:59.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 -- 172.21.15.93:0/120326810 shutdown_connections 2021-11-20T09:00:59.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.252+0000 7fe95273c700 1 -- 172.21.15.93:0/120326810 wait complete. 2021-11-20T09:00:59.255 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe95273c700 1 Processor -- start 2021-11-20T09:00:59.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe95273c700 1 -- start start 2021-11-20T09:00:59.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe95273c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe94c0f6680 0x7fe94c109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:59.256 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe95273c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:59.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe95273c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c109b20 con 0x7fe94c0f7920 2021-11-20T09:00:59.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe95273c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c109c60 con 0x7fe94c0f6680 2021-11-20T09:00:59.257 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:59.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44548/0 (socket says 172.21.15.93:44548) 2021-11-20T09:00:59.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94b7fe700 1 -- 172.21.15.93:0/463601698 learned_addr learned my addr 172.21.15.93:0/463601698 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:00:59.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94bfff700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe94c0f6680 0x7fe94c109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:59.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94b7fe700 1 -- 172.21.15.93:0/463601698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe94c0f6680 msgr2=0x7fe94c109080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:59.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94b7fe700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe94c0f6680 0x7fe94c109080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:59.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94b7fe700 1 -- 172.21.15.93:0/463601698 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940005040 con 0x7fe94c0f7920 2021-11-20T09:00:59.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.253+0000 7fe94b7fe700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c109570 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fe940009830 tx=0x7fe940009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:59.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.254+0000 7fe948ff9700 1 -- 172.21.15.93:0/463601698 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe940018070 con 0x7fe94c0f7920 2021-11-20T09:00:59.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.254+0000 7fe948ff9700 1 -- 172.21.15.93:0/463601698 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe940013410 con 0x7fe94c0f7920 2021-11-20T09:00:59.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.254+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c10e6a0 con 0x7fe94c0f7920 2021-11-20T09:00:59.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.254+0000 7fe948ff9700 1 -- 172.21.15.93:0/463601698 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe94001c400 con 0x7fe94c0f7920 2021-11-20T09:00:59.263 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.254+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c10ec60 con 0x7fe94c0f7920 2021-11-20T09:00:59.263 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.255+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c027a00 con 0x7fe94c0f7920 2021-11-20T09:00:59.263 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.255+0000 7fe948ff9700 1 -- 172.21.15.93:0/463601698 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe94001d420 con 0x7fe94c0f7920 2021-11-20T09:00:59.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.255+0000 7fe948ff9700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe93405c2a0 0x7fe93405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:00:59.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.255+0000 7fe948ff9700 1 -- 172.21.15.93:0/463601698 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe940078220 con 0x7fe94c0f7920 2021-11-20T09:00:59.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.255+0000 7fe94bfff700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe93405c2a0 0x7fe93405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:00:59.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.256+0000 7fe94bfff700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe93405c2a0 0x7fe93405e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fe94c0f7780 tx=0x7fe93c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:00:59.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.259+0000 7fe948ff9700 1 -- 172.21.15.93:0/463601698 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe94004aa30 con 0x7fe94c0f7920 2021-11-20T09:00:59.441 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.443+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe94c0fee10 con 0x7fe94c0f7920 2021-11-20T09:00:59.442 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.443+0000 7fe948ff9700 1 -- 172.21.15.93:0/463601698 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fe94004e050 con 0x7fe94c0f7920 2021-11-20T09:00:59.444 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:00:59.445 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:00:59.447 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe93405c2a0 msgr2=0x7fe93405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:59.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe93405c2a0 0x7fe93405e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fe94c0f7780 tx=0x7fe93c009040).stop 2021-11-20T09:00:59.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 msgr2=0x7fe94c109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:00:59.448 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c109570 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fe940009830 tx=0x7fe940009b30).stop 2021-11-20T09:00:59.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 shutdown_connections 2021-11-20T09:00:59.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe93405c2a0 0x7fe93405e760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:59.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe94c0f6680 0x7fe94c109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:59.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 --2- 172.21.15.93:0/463601698 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe94c0f7920 0x7fe94c109570 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:00:59.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 >> 172.21.15.93:0/463601698 conn(0x7fe94c0f1de0 msgr2=0x7fe94c06a520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:00:59.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 shutdown_connections 2021-11-20T09:00:59.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:00:59.445+0000 7fe95273c700 1 -- 172.21.15.93:0/463601698 wait complete. 2021-11-20T09:00:59.452 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:00:59.456 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:00:59.456 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:01:00.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:00:59 smithi102 conmon[36472]: audit 2021-11-20T09:00:59.444190+0000 mon.smithi093 (mon.0) 850 : audit [DBG] from='client.? 172.21.15.93:0/463601698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:00.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:59 smithi093 conmon[33522]: audit 2021-11-20T09:00:59.444190+0000 mon.smithi093 ( 2021-11-20T09:01:00.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:00:59 smithi093 conmon[33522]: mon.0) 850 : audit [DBG] from='client.? 172.21.15.93:0/463601698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:00.459 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:00.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1680517879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f7920 msgr2=0x7ff4b00f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:00.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1680517879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f7920 0x7ff4b00f7d80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ff498004660 tx=0x7ff498009b30).stop 2021-11-20T09:01:00.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1680517879 shutdown_connections 2021-11-20T09:01:00.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1680517879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f7920 0x7ff4b00f7d80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:00.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1680517879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4b00f6680 0x7ff4b00f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:00.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1680517879 >> 172.21.15.93:0/1680517879 conn(0x7ff4b00f1de0 msgr2=0x7ff4b00f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:00.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1680517879 shutdown_connections 2021-11-20T09:01:00.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.584+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1680517879 wait complete. 2021-11-20T09:01:00.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4b57bc700 1 Processor -- start 2021-11-20T09:01:00.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4b57bc700 1 -- start start 2021-11-20T09:01:00.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4b57bc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f6680 0x7ff4b0106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:00.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4b57bc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4b00f7920 0x7ff4b0107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:00.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4b57bc700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b0107930 con 0x7ff4b00f6680 2021-11-20T09:01:00.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4b57bc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b0107a70 con 0x7ff4b00f7920 2021-11-20T09:01:00.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4aeffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f6680 0x7ff4b0106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:00.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4aeffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f6680 0x7ff4b0106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44556/0 (socket says 172.21.15.93:44556) 2021-11-20T09:01:00.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4aeffd700 1 -- 172.21.15.93:0/1895534280 learned_addr learned my addr 172.21.15.93:0/1895534280 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:00.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4aeffd700 1 -- 172.21.15.93:0/1895534280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4b00f7920 msgr2=0x7ff4b0107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:00.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4a67fc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4b00f7920 0x7ff4b0107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:00.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4aeffd700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4b00f7920 0x7ff4b0107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:00.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4aeffd700 1 -- 172.21.15.93:0/1895534280 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff498005040 con 0x7ff4b00f6680 2021-11-20T09:01:00.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4a67fc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4b00f7920 0x7ff4b0107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:01:00.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.585+0000 7ff4aeffd700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f6680 0x7ff4b0106e90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7ff4a000c530 tx=0x7ff4a0007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:00.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.586+0000 7ff4a7fff700 1 -- 172.21.15.93:0/1895534280 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a000f650 con 0x7ff4b00f6680 2021-11-20T09:01:00.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.586+0000 7ff4a7fff700 1 -- 172.21.15.93:0/1895534280 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4a000fc60 con 0x7ff4b00f6680 2021-11-20T09:01:00.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.586+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b010c510 con 0x7ff4b00f6680 2021-11-20T09:01:00.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.586+0000 7ff4a7fff700 1 -- 172.21.15.93:0/1895534280 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a0012d20 con 0x7ff4b00f6680 2021-11-20T09:01:00.595 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.586+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b010cad0 con 0x7ff4b00f6680 2021-11-20T09:01:00.595 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.587+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b0027a00 con 0x7ff4b00f6680 2021-11-20T09:01:00.595 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.587+0000 7ff4a7fff700 1 -- 172.21.15.93:0/1895534280 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff4a0019030 con 0x7ff4b00f6680 2021-11-20T09:01:00.596 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.588+0000 7ff4a7fff700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff49c05c2f0 0x7ff49c05e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:00.596 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.588+0000 7ff4a7fff700 1 -- 172.21.15.93:0/1895534280 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6586+0+0 (secure 0 0 0) 0x7ff4a0076580 con 0x7ff4b00f6680 2021-11-20T09:01:00.597 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.588+0000 7ff4a67fc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff49c05c2f0 0x7ff49c05e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:00.597 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.588+0000 7ff4a67fc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff49c05c2f0 0x7ff49c05e7b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7ff498009d60 tx=0x7ff498009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:00.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.591+0000 7ff4a7fff700 1 -- 172.21.15.93:0/1895534280 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4a0048de0 con 0x7ff4b00f6680 2021-11-20T09:01:00.775 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.775+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4b00fee10 con 0x7ff4b00f6680 2021-11-20T09:01:00.776 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.775+0000 7ff4a7fff700 1 -- 172.21.15.93:0/1895534280 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+2188 (secure 0 0 0) 0x7ff4a0017090 con 0x7ff4b00f6680 2021-11-20T09:01:00.776 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:00.776 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:00:46.681933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":81,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/938076909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":938076909},{"type":"v1","addr":"172.21.15.102:6827","nonce":938076909}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:00.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff49c05c2f0 msgr2=0x7ff49c05e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:00.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff49c05c2f0 0x7ff49c05e7b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7ff498009d60 tx=0x7ff498009b30).stop 2021-11-20T09:01:00.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f6680 msgr2=0x7ff4b0106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:00.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f6680 0x7ff4b0106e90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7ff4a000c530 tx=0x7ff4a0007e10).stop 2021-11-20T09:01:00.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 shutdown_connections 2021-11-20T09:01:00.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff49c05c2f0 0x7ff49c05e7b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:00.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4b00f6680 0x7ff4b0106e90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:00.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 --2- 172.21.15.93:0/1895534280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4b00f7920 0x7ff4b0107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:00.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 >> 172.21.15.93:0/1895534280 conn(0x7ff4b00f1de0 msgr2=0x7ff4b00fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:00.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 shutdown_connections 2021-11-20T09:01:00.781 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:00.777+0000 7ff4b57bc700 1 -- 172.21.15.93:0/1895534280 wait complete. 2021-11-20T09:01:00.781 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 85 2021-11-20T09:01:00.788 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:00 smithi093 conmon[33522]: cluster 2021-11-20T09:00:59.583427+0000 mgr.smithi093.qdikkm 2021-11-20T09:01:00.788 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:00 smithi093 conmon[33522]: (mgr.14162) 313 : cluster [DBG] pgmap v290: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-20T09:01:00.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:00:46.681933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 81, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/938076909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 938076909}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 938076909}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:00.792 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24521:up:clientreplay 2021-11-20T09:01:01.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:00 smithi102 conmon[36472]: cluster 2021-11-20T09:00:59.583427+0000 mgr.smithi093.qdikkm (mgr.14162) 313 : cluster [DBG] 2021-11-20T09:01:01.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:00 smithi102 conmon[36472]: pgmap v290: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-20T09:01:01.794 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:01.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:01 smithi102 conmon[36472]: audit 2021-11-20T09:01:00.777583+0000 mon.smithi093 (mon.0) 851 : audit [DBG] from='client.? 172.21.15.93:0/1895534280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:01.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:01 smithi102 conmon[36472]: cluster 2021-11-20T09:01:01.162481+0000 mon.smithi093 (mon.0) 852 : cluster [INF] daemon mds.cephfs.smithi102.hjodmi restarted 2021-11-20T09:01:01.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:01 smithi102 conmon[36472]: cluster 2021-11-20T09:01:01.166831+0000 mon.smithi093 (mon.0) 853 : cluster [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T09:01:01.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:01 smithi102 conmon[36472]: cluster 2021-11-20T09:01:01.604899+0000 mon.smithi093 (mon.0) 854 : cluster [INF] Standby daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 2021-11-20T09:01:01.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:01 smithi102 conmon[36472]: cluster 2021-11-20T09:01:01.608744+0000 mon.smithi093 (mon.0) 855 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:boot 2021-11-20T09:01:01.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:01 smithi102 conmon[36472]: cluster 2021-11-20T09:01:01.608826+0000 mon.smithi093 (mon.0) 856 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:replay} 1 up:standby 2021-11-20T09:01:01.921 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:01 smithi102 conmon[36472]: audit 2021-11-20T09:01:01.609113+0000 mon.smithi093 (mon.0) 857 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T09:01:01.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 -- 172.21.15.93:0/2141188462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 msgr2=0x7f7e640f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:01.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 --2- 172.21.15.93:0/2141188462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e640f7cd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f7e58004660 tx=0x7f7e58009b30).stop 2021-11-20T09:01:01.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 -- 172.21.15.93:0/2141188462 shutdown_connections 2021-11-20T09:01:01.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 --2- 172.21.15.93:0/2141188462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e640f7cd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:01.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 --2- 172.21.15.93:0/2141188462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e640f65d0 0x7f7e640f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:01.931 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 -- 172.21.15.93:0/2141188462 >> 172.21.15.93:0/2141188462 conn(0x7f7e640f1d10 msgr2=0x7f7e640f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:01.931 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 -- 172.21.15.93:0/2141188462 shutdown_connections 2021-11-20T09:01:01.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.918+0000 7f7e6880a700 1 -- 172.21.15.93:0/2141188462 wait complete. 2021-11-20T09:01:01.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e6880a700 1 Processor -- start 2021-11-20T09:01:01.933 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e6880a700 1 -- start start 2021-11-20T09:01:01.933 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e6880a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e640f65d0 0x7f7e6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:01.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e6880a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e64069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:01.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e6880a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6406a410 con 0x7f7e640f7870 2021-11-20T09:01:01.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e6880a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6406a550 con 0x7f7e640f65d0 2021-11-20T09:01:01.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e62ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e64069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:01.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e62ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e64069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44566/0 (socket says 172.21.15.93:44566) 2021-11-20T09:01:01.936 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e62ffd700 1 -- 172.21.15.93:0/903146600 learned_addr learned my addr 172.21.15.93:0/903146600 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:01.936 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.919+0000 7f7e637fe700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e640f65d0 0x7f7e6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:01.937 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e62ffd700 1 -- 172.21.15.93:0/903146600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e640f65d0 msgr2=0x7f7e6406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:01.937 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e62ffd700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e640f65d0 0x7f7e6406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:01.937 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e62ffd700 1 -- 172.21.15.93:0/903146600 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58005040 con 0x7f7e640f7870 2021-11-20T09:01:01.938 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e62ffd700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e64069e20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f7e58009830 tx=0x7f7e58009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:01.938 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e4bfff700 1 -- 172.21.15.93:0/903146600 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e58018070 con 0x7f7e640f7870 2021-11-20T09:01:01.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6406a7d0 con 0x7f7e640f7870 2021-11-20T09:01:01.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e4bfff700 1 -- 172.21.15.93:0/903146600 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e58013410 con 0x7f7e640f7870 2021-11-20T09:01:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e4bfff700 1 -- 172.21.15.93:0/903146600 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e5801c400 con 0x7f7e640f7870 2021-11-20T09:01:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.920+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6410ed60 con 0x7f7e640f7870 2021-11-20T09:01:01.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.921+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e64045ad0 con 0x7f7e640f7870 2021-11-20T09:01:01.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.921+0000 7f7e4bfff700 1 -- 172.21.15.93:0/903146600 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7e5801d420 con 0x7f7e640f7870 2021-11-20T09:01:01.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.921+0000 7f7e4bfff700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e4c05c300 0x7f7e4c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:01.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.921+0000 7f7e4bfff700 1 -- 172.21.15.93:0/903146600 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f7e58078270 con 0x7f7e640f7870 2021-11-20T09:01:01.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.922+0000 7f7e637fe700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e4c05c300 0x7f7e4c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:01.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.922+0000 7f7e637fe700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e4c05c300 0x7f7e4c05e7c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7e6406b730 tx=0x7f7e54009220).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:01.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:01.925+0000 7f7e4bfff700 1 -- 172.21.15.93:0/903146600 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e5804aa30 con 0x7f7e640f7870 2021-11-20T09:01:02.107 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.107+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e64027a00 con 0x7f7e640f7870 2021-11-20T09:01:02.108 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.108+0000 7f7e4bfff700 1 -- 172.21.15.93:0/903146600 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2186 (secure 0 0 0) 0x7f7e5804e050 con 0x7f7e640f7870 2021-11-20T09:01:02.109 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:02.110 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":86}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:01.605035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:replay","state_seq":2,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:02.112 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e4c05c300 msgr2=0x7f7e4c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:02.114 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e4c05c300 0x7f7e4c05e7c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7e6406b730 tx=0x7f7e54009220).stop 2021-11-20T09:01:02.114 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 msgr2=0x7f7e64069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:02.114 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e64069e20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f7e58009830 tx=0x7f7e58009b30).stop 2021-11-20T09:01:02.115 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 shutdown_connections 2021-11-20T09:01:02.116 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7e4c05c300 0x7f7e4c05e7c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:02.116 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e640f65d0 0x7f7e6406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:02.117 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 --2- 172.21.15.93:0/903146600 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e640f7870 0x7f7e64069e20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:02.117 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 >> 172.21.15.93:0/903146600 conn(0x7f7e640f1d10 msgr2=0x7f7e640fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:02.117 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 shutdown_connections 2021-11-20T09:01:02.117 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:02.110+0000 7f7e6880a700 1 -- 172.21.15.93:0/903146600 wait complete. 2021-11-20T09:01:02.118 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 86 2021-11-20T09:01:02.121 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:01 smithi093 conmon[33522]: audit 2021-11-20T09:01:00.777583+0000 mon.smithi093 (mon.0) 851 : audit [DBG] from='client.? 172.21.15.93:0/1895534280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:02.121 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:01 smithi093 conmon[33522]: cluster 2021-11-20T09:01:01.162481+0000 mon.smithi093 (mon.0) 852 : cluster [INF] daemon mds.cephfs.smithi102.hjodmi restarted 2021-11-20T09:01:02.122 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:01 smithi093 conmon[33522]: cluster 2021-11-20T09:01:01.166831+0000 mon.smithi093 (mon.0) 853 : cluster [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T09:01:02.122 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:01 smithi093 conmon[33522]: cluster 2021-11-20T09:01:01.604899+0000 mon.smithi093 (mon.0) 854 : cluster [INF] Standby daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 2021-11-20T09:01:02.122 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:01 smithi093 conmon[33522]: cluster 2021-11-20T09:01:01.608744+0000 mon.smithi093 (mon.0) 855 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:boot 2021-11-20T09:01:02.123 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:01 smithi093 conmon[33522]: cluster 2021-11-20T09:01:01.608826+0000 mon.smithi093 (mon.0) 856 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:replay} 1 up:standby 2021-11-20T09:01:02.123 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:01 smithi093 conmon[33522]: audit 2021-11-20T09:01:01.609113+0000 mon.smithi093 (mon.0) 857 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T09:01:02.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:01.605035+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:02.125 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:replay 2021-11-20T09:01:02.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:02 smithi093 conmon[33522]: cluster 2021-11-20T09:01:01.583836+0000 mgr.smithi093.qdikkm (mgr.14162) 314 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:01:02.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:02 smithi093 conmon[33522]: audit 2021-11-20T09: 2021-11-20T09:01:02.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:02 smithi093 conmon[33522]: 01:02.109921+0000 mon.smithi093 (mon.0) 858 : audit [DBG] from='client.? 172.21.15.93:0/903146600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:02.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:02 smithi093 conmon[33522]: cluster 2021-11-20T09:01:02.613596+0000 mon.smithi093 (mon.0) 859 : cluster [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:reconnect 2021-11-20T09:01:02.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:02 smithi093 conmon[33522]: cluster 2021-11-20T 2021-11-20T09:01:02.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:02 smithi093 conmon[33522]: 09:01:02.613652+0000 mon.smithi093 (mon.0) 860 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:reconnect} 1 up:standby 2021-11-20T09:01:03.126 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:03.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:02 smithi102 conmon[36472]: cluster 2021-11-20T09:01:01.583836+0000 mgr.smithi093.qdikkm (mgr.14162) 314 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T09:01:03.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:02 smithi102 conmon[36472]: audit 2021-11-20T09:01:02.109921+0000 mon.smithi093 (mon.0) 858 : audit [DBG] from='client.? 172.21.15.93:0/903146600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:03.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:02 smithi102 conmon[36472]: cluster 2021-11-20T09:01:02.613596+0000 mon.smithi093 (mon.0) 859 : cluster [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:reconnect 2021-11-20T09:01:03.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:02 smithi102 conmon[36472]: cluster 2021-11-20T09:01:02.613652+0000 mon.smithi093 (mon.0) 860 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:reconnect} 1 up:standby 2021-11-20T09:01:03.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 -- 172.21.15.93:0/3434501628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 msgr2=0x7fc8640f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:03.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 --2- 172.21.15.93:0/3434501628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8640f7da0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fc854004660 tx=0x7fc854009b30).stop 2021-11-20T09:01:03.258 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 -- 172.21.15.93:0/3434501628 shutdown_connections 2021-11-20T09:01:03.259 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 --2- 172.21.15.93:0/3434501628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8640f7da0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:03.259 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 --2- 172.21.15.93:0/3434501628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8640f66a0 0x7fc8640f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:03.259 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 -- 172.21.15.93:0/3434501628 >> 172.21.15.93:0/3434501628 conn(0x7fc8640f1da0 msgr2=0x7fc8640f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:03.260 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 -- 172.21.15.93:0/3434501628 shutdown_connections 2021-11-20T09:01:03.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 -- 172.21.15.93:0/3434501628 wait complete. 2021-11-20T09:01:03.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 Processor -- start 2021-11-20T09:01:03.261 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.254+0000 7fc86c387700 1 -- start start 2021-11-20T09:01:03.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc86c387700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8640f66a0 0x7fc864106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:03.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc86c387700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8641073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:03.262 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc86c387700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc864107960 con 0x7fc8640f7940 2021-11-20T09:01:03.263 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc86c387700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc864107aa0 con 0x7fc8640f66a0 2021-11-20T09:01:03.263 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc869922700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8641073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:03.263 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc869922700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8641073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44576/0 (socket says 172.21.15.93:44576) 2021-11-20T09:01:03.263 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc869922700 1 -- 172.21.15.93:0/2013244858 learned_addr learned my addr 172.21.15.93:0/2013244858 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:03.264 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc869922700 1 -- 172.21.15.93:0/2013244858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8640f66a0 msgr2=0x7fc864106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:03.264 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc869922700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8640f66a0 0x7fc864106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:03.264 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc869922700 1 -- 172.21.15.93:0/2013244858 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc854005040 con 0x7fc8640f7940 2021-11-20T09:01:03.264 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc869922700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8641073b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fc854009f50 tx=0x7fc854009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:03.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc85affd700 1 -- 172.21.15.93:0/2013244858 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc854018070 con 0x7fc8640f7940 2021-11-20T09:01:03.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc85affd700 1 -- 172.21.15.93:0/2013244858 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc854013410 con 0x7fc8640f7940 2021-11-20T09:01:03.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86410c4e0 con 0x7fc8640f7940 2021-11-20T09:01:03.265 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc85affd700 1 -- 172.21.15.93:0/2013244858 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc85401c3c0 con 0x7fc8640f7940 2021-11-20T09:01:03.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.255+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86410caa0 con 0x7fc8640f7940 2021-11-20T09:01:03.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.257+0000 7fc85affd700 1 -- 172.21.15.93:0/2013244858 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc85401d420 con 0x7fc8640f7940 2021-11-20T09:01:03.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.257+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc864027a00 con 0x7fc8640f7940 2021-11-20T09:01:03.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.257+0000 7fc85affd700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc85005c2a0 0x7fc85005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:03.267 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.257+0000 7fc85affd700 1 -- 172.21.15.93:0/2013244858 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc854077e80 con 0x7fc8640f7940 2021-11-20T09:01:03.267 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.257+0000 7fc86a123700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc85005c2a0 0x7fc85005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:03.267 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.257+0000 7fc86a123700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc85005c2a0 0x7fc85005e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc8640f77a0 tx=0x7fc8600099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:03.268 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.261+0000 7fc85affd700 1 -- 172.21.15.93:0/2013244858 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc85404a6c0 con 0x7fc8640f7940 2021-11-20T09:01:03.444 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.444+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8640fee30 con 0x7fc8640f7940 2021-11-20T09:01:03.445 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.445+0000 7fc85affd700 1 -- 172.21.15.93:0/2013244858 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2189 (secure 0 0 0) 0x7fc85404dce0 con 0x7fc8640f7940 2021-11-20T09:01:03.446 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:03.447 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":86}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:02.610160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:03.449 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc85005c2a0 msgr2=0x7fc85005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:03.450 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc85005c2a0 0x7fc85005e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc8640f77a0 tx=0x7fc8600099e0).stop 2021-11-20T09:01:03.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 msgr2=0x7fc8641073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:03.451 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8641073b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fc854009f50 tx=0x7fc854009b30).stop 2021-11-20T09:01:03.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 shutdown_connections 2021-11-20T09:01:03.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc85005c2a0 0x7fc85005e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:03.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8640f66a0 0x7fc864106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:03.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 --2- 172.21.15.93:0/2013244858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8640f7940 0x7fc8641073b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:03.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 >> 172.21.15.93:0/2013244858 conn(0x7fc8640f1da0 msgr2=0x7fc8640fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:03.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 shutdown_connections 2021-11-20T09:01:03.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:03.446+0000 7fc86c387700 1 -- 172.21.15.93:0/2013244858 wait complete. 2021-11-20T09:01:03.453 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 87 2021-11-20T09:01:03.460 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:02.610160+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:03.461 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:reconnect 2021-11-20T09:01:04.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: cluster 2021-11-20T09:01:02.633417+0000 mds.cephfs.smithi102.lhrlmh (mds.0) 2021-11-20T09:01:04.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:01:04.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: audit 2021-11-20T09:01:03.446864+0000 mon.smithi093 (mon.0) 861 : audit [DBG] from='client.? 172.21.15.93:0/2013244858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:04.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: cluster 2021-11-20T09:01:03.615154+0000 mon.smithi093 (mon.0) 862 : cluster [DBG] 2021-11-20T09:01:04.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:rejoin 2021-11-20T09:01:04.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: cluster 2021- 2021-11-20T09:01:04.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: 11-20T09:01:03.615203+0000 mon.smithi093 (mon.0) 863 2021-11-20T09:01:04.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:03 smithi102 conmon[36472]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:rejoin} 1 up:standby 2021-11-20T09:01:04.204 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: cluster 2021-11-20T09:01:02.633417+0000 mds.cephfs.smithi102.lhrlmh (mds.0) 2021-11-20T09:01:04.205 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:01:04.206 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: audit 2021-11-20T09:01:03.446864+0000 mon.smithi093 (mon.0) 861 : audit [DBG] from='client.? 172.21.15.93:0/2013244858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:04.206 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: cluster 2021-11-20T09:01:03.615154+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:04.207 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: 862 : cluster [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:rejoin 2021-11-20T09:01:04.207 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: cluster 2021-11-20T09: 2021-11-20T09:01:04.207 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: 01:03.615203+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:04.208 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:03 smithi093 conmon[33522]: 863 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:rejoin} 1 up:standby 2021-11-20T09:01:04.463 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:04.584 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 -- 172.21.15.93:0/2390512860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 msgr2=0x7f1d140f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:04.584 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/2390512860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d140f6ae0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f1d08004660 tx=0x7f1d08009b30).stop 2021-11-20T09:01:04.584 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 -- 172.21.15.93:0/2390512860 shutdown_connections 2021-11-20T09:01:04.585 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/2390512860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140f7960 0x7f1d140f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:04.585 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/2390512860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d140f6ae0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:04.585 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 -- 172.21.15.93:0/2390512860 >> 172.21.15.93:0/2390512860 conn(0x7f1d140f1de0 msgr2=0x7f1d140f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:04.585 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 -- 172.21.15.93:0/2390512860 shutdown_connections 2021-11-20T09:01:04.585 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.579+0000 7f1d1a18f700 1 -- 172.21.15.93:0/2390512860 wait complete. 2021-11-20T09:01:04.586 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d1a18f700 1 Processor -- start 2021-11-20T09:01:04.586 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d1a18f700 1 -- start start 2021-11-20T09:01:04.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d1a18f700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d14106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:04.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d1a18f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140f7960 0x7f1d141073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:04.587 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d1a18f700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14107960 con 0x7f1d140f66c0 2021-11-20T09:01:04.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d1a18f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14107aa0 con 0x7f1d140f7960 2021-11-20T09:01:04.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d137fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d14106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:04.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d137fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d14106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44584/0 (socket says 172.21.15.93:44584) 2021-11-20T09:01:04.588 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d137fe700 1 -- 172.21.15.93:0/1127116223 learned_addr learned my addr 172.21.15.93:0/1127116223 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:04.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.580+0000 7f1d12ffd700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140f7960 0x7f1d141073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:04.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1d137fe700 1 -- 172.21.15.93:0/1127116223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140f7960 msgr2=0x7f1d141073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:04.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1d137fe700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140f7960 0x7f1d141073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:04.589 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1d137fe700 1 -- 172.21.15.93:0/1127116223 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08005040 con 0x7f1d140f66c0 2021-11-20T09:01:04.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1d137fe700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d14106ec0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f1d08004e30 tx=0x7f1d08009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:04.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1cfbfff700 1 -- 172.21.15.93:0/1127116223 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d08018070 con 0x7f1d140f66c0 2021-11-20T09:01:04.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1cfbfff700 1 -- 172.21.15.93:0/1127116223 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d08013410 con 0x7f1d140f66c0 2021-11-20T09:01:04.590 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1cfbfff700 1 -- 172.21.15.93:0/1127116223 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d0801de20 con 0x7f1d140f66c0 2021-11-20T09:01:04.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1410c4e0 con 0x7f1d140f66c0 2021-11-20T09:01:04.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.581+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1410caa0 con 0x7f1d140f66c0 2021-11-20T09:01:04.591 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.582+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d14027a00 con 0x7f1d140f66c0 2021-11-20T09:01:04.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.582+0000 7f1cfbfff700 1 -- 172.21.15.93:0/1127116223 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1d0801d420 con 0x7f1d140f66c0 2021-11-20T09:01:04.592 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.582+0000 7f1cfbfff700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1cfc05c2a0 0x7f1cfc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:04.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.583+0000 7f1cfbfff700 1 -- 172.21.15.93:0/1127116223 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f1d08077f20 con 0x7f1d140f66c0 2021-11-20T09:01:04.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.583+0000 7f1d12ffd700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1cfc05c2a0 0x7f1cfc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:04.593 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.583+0000 7f1d12ffd700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1cfc05c2a0 0x7f1cfc05e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1d04004c40 tx=0x7f1d0400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:04.594 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.586+0000 7f1cfbfff700 1 -- 172.21.15.93:0/1127116223 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d080a42b0 con 0x7f1d140f66c0 2021-11-20T09:01:04.773 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.773+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d140fee50 con 0x7f1d140f66c0 2021-11-20T09:01:04.774 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.773+0000 7f1cfbfff700 1 -- 172.21.15.93:0/1127116223 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f1d0804a760 con 0x7f1d140f66c0 2021-11-20T09:01:04.775 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:04.776 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":86}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:04.778 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1cfc05c2a0 msgr2=0x7f1cfc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:04.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1cfc05c2a0 0x7f1cfc05e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1d04004c40 tx=0x7f1d0400b040).stop 2021-11-20T09:01:04.779 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 msgr2=0x7f1d14106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:04.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d14106ec0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f1d08004e30 tx=0x7f1d08009b30).stop 2021-11-20T09:01:04.780 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 shutdown_connections 2021-11-20T09:01:04.781 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f1cfc05c2a0 0x7f1cfc05e760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:04.781 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140f66c0 0x7f1d14106ec0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:04.782 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 --2- 172.21.15.93:0/1127116223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d140f7960 0x7f1d141073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:04.782 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 >> 172.21.15.93:0/1127116223 conn(0x7f1d140f1de0 msgr2=0x7f1d140fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:04.782 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 shutdown_connections 2021-11-20T09:01:04.783 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:04.775+0000 7f1d1a18f700 1 -- 172.21.15.93:0/1127116223 wait complete. 2021-11-20T09:01:04.783 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 89 2021-11-20T09:01:04.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:04.792 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:04.918 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:04 smithi102 conmon[36472]: cluster 2021-11-20T09:01:03.584427+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T09:01:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:04 smithi102 conmon[36472]: ) 315 : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T09:01:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:04 smithi102 conmon[36472]: audit 2021-11-20T09:01:03.975865+0000 mon.smithi093 (mon.0) 864 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T09:01:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:04 smithi102 conmon[36472]: audit 2021-11-20T09:01:04.355958+0000 mon.smithi093 (mon.0) 865 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T09:01:04.919 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:04 smithi102 conmon[36472]: cluster 2021-11-20T09:01:04.618113+0000 mon.smithi093 (mon.0) 866 : cluster [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:clientreplay 2021-11-20T09:01:04.920 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:04 smithi102 conmon[36472]: cluster 2021-11-20T09:01:04.618181+0000 mon.smithi093 (mon.0) 867 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:05.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:04 smithi093 conmon[33522]: cluster 2021-11-20T09:01:03.584427+0000 mgr.smithi093.qdikkm (mgr.14162) 315 2021-11-20T09:01:05.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:04 smithi093 conmon[33522]: : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T09:01:05.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:04 smithi093 conmon[33522]: audit 2021-11-20T09:01:03.975865+0000 mon.smithi093 (mon.0) 864 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T09:01:05.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:04 smithi093 conmon[33522]: audit 2021-11-20T09:01:04.355958+0000 mon.smithi093 (mon.0) 865 : audit [INF] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' 2021-11-20T09:01:05.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:04 smithi093 conmon[33522]: cluster 2021-11-20T09:01:04.618113+0000 mon.smithi093 (mon.0) 866 : cluster [DBG] mds.? [v2:172.21.15.102:6824/2583579411,v1:172.21.15.102:6825/2583579411] up:clientreplay 2021-11-20T09:01:05.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:04 smithi093 conmon[33522]: cluster 2021-11-20T09:01:04.618181+0000 mon.smithi093 (mon.0) 867 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:05.793 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:05.926 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 -- 172.21.15.93:0/6804170 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 msgr2=0x7f7ee00f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:05.927 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/6804170 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee00f6ca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7ec8004660 tx=0x7f7ec8009b30).stop 2021-11-20T09:01:05.927 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 -- 172.21.15.93:0/6804170 shutdown_connections 2021-11-20T09:01:05.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/6804170 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee00f6ca0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:05.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/6804170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ee00f55a0 0x7f7ee00f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:05.928 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 -- 172.21.15.93:0/6804170 >> 172.21.15.93:0/6804170 conn(0x7f7ee00f0ce0 msgr2=0x7f7ee00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:05.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 -- 172.21.15.93:0/6804170 shutdown_connections 2021-11-20T09:01:05.929 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 -- 172.21.15.93:0/6804170 wait complete. 2021-11-20T09:01:05.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 Processor -- start 2021-11-20T09:01:05.930 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.919+0000 7f7ee58b6700 1 -- start start 2021-11-20T09:01:05.931 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ee58b6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ee00f55a0 0x7f7ee005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:05.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ee58b6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee010e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:05.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ee58b6700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee0107a70 con 0x7f7ee00f6840 2021-11-20T09:01:05.932 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ee58b6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee010e580 con 0x7f7ee00f55a0 2021-11-20T09:01:05.933 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ede7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee010e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:05.933 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ede7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee010e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44598/0 (socket says 172.21.15.93:44598) 2021-11-20T09:01:05.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ede7fc700 1 -- 172.21.15.93:0/2667712843 learned_addr learned my addr 172.21.15.93:0/2667712843 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:05.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7edeffd700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ee00f55a0 0x7f7ee005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:05.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ede7fc700 1 -- 172.21.15.93:0/2667712843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ee00f55a0 msgr2=0x7f7ee005ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:05.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ede7fc700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ee00f55a0 0x7f7ee005ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:05.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ede7fc700 1 -- 172.21.15.93:0/2667712843 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8005040 con 0x7f7ee00f6840 2021-11-20T09:01:05.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7edeffd700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ee00f55a0 0x7f7ee005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:01:05.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ede7fc700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee010e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f7ec8009830 tx=0x7f7ec8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:05.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ed77fe700 1 -- 172.21.15.93:0/2667712843 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec8018070 con 0x7f7ee00f6840 2021-11-20T09:01:05.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ed77fe700 1 -- 172.21.15.93:0/2667712843 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ec8013410 con 0x7f7ee00f6840 2021-11-20T09:01:05.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee010e6c0 con 0x7f7ee00f6840 2021-11-20T09:01:05.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ed77fe700 1 -- 172.21.15.93:0/2667712843 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec801c3c0 con 0x7f7ee00f6840 2021-11-20T09:01:05.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.920+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee010ec20 con 0x7f7ee00f6840 2021-11-20T09:01:05.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.921+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee0027a00 con 0x7f7ee00f6840 2021-11-20T09:01:05.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.922+0000 7f7ed77fe700 1 -- 172.21.15.93:0/2667712843 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7ec801d420 con 0x7f7ee00f6840 2021-11-20T09:01:05.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.922+0000 7f7ed77fe700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ecc05c2a0 0x7f7ecc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:05.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.922+0000 7f7ed77fe700 1 -- 172.21.15.93:0/2667712843 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f7ec8078130 con 0x7f7ee00f6840 2021-11-20T09:01:05.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.922+0000 7f7edeffd700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ecc05c2a0 0x7f7ecc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:05.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.922+0000 7f7edeffd700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ecc05c2a0 0x7f7ecc05e760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f7ee00f66a0 tx=0x7f7ed0004d00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:05.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:05.926+0000 7f7ed77fe700 1 -- 172.21.15.93:0/2667712843 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ec804a8f0 con 0x7f7ee00f6840 2021-11-20T09:01:06.030 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:05 smithi093 conmon[33522]: audit 2021-11-20T09:01:04.775561+0000 mon.smithi093 (mon.0) 868 : audit [DBG] from='client.? 172.21.15.93:0/1127116223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:06.116 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.116+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ee00fdd30 con 0x7f7ee00f6840 2021-11-20T09:01:06.117 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.116+0000 7f7ed77fe700 1 -- 172.21.15.93:0/2667712843 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f7ec804df10 con 0x7f7ee00f6840 2021-11-20T09:01:06.117 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:06.118 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:06.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ecc05c2a0 msgr2=0x7f7ecc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:06.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ecc05c2a0 0x7f7ecc05e760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f7ee00f66a0 tx=0x7f7ed0004d00).stop 2021-11-20T09:01:06.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 msgr2=0x7f7ee010e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:06.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee010e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f7ec8009830 tx=0x7f7ec8009b30).stop 2021-11-20T09:01:06.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 shutdown_connections 2021-11-20T09:01:06.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f7ecc05c2a0 0x7f7ecc05e760 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:06.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ee00f55a0 0x7f7ee005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:06.127 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 --2- 172.21.15.93:0/2667712843 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee00f6840 0x7f7ee010e090 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:06.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 >> 172.21.15.93:0/2667712843 conn(0x7f7ee00f0ce0 msgr2=0x7f7ee00fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:06.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 shutdown_connections 2021-11-20T09:01:06.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:06.118+0000 7f7ee58b6700 1 -- 172.21.15.93:0/2667712843 wait complete. 2021-11-20T09:01:06.128 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:06.130 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:06.130 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:06.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:05 smithi102 conmon[36472]: audit 2021-11-20T09:01:04.775561+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:06.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:05 smithi102 conmon[36472]: 868 : audit [DBG] from='client.? 172.21.15.93:0/1127116223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:07.132 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:07.158 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: cluster 2021-11-20T09:01:05.585032+0000 2021-11-20T09:01:07.159 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 316 : cluster [DBG] pgmap v294: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T09:01:07.159 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: cluster 2021-11-20T09:01:05.755740+0000 mon.smithi093 (mon.0) 869 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:standby 2021-11-20T09:01:07.159 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: cluster 2021-11-20T09:01:05.755824+0000 mon.smithi093 2021-11-20T09:01:07.159 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: (mon.0) 870 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:07.160 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: audit 2021- 2021-11-20T09:01:07.160 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: 11-20T09:01:06.118560+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:07.160 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:06 smithi093 conmon[33522]: 871 : audit [DBG] from='client.? 172.21.15.93:0/2667712843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:07.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:06 smithi102 conmon[36472]: cluster 2021-11-20T09:01:05.585032+0000 mgr.smithi093.qdikkm (mgr.14162) 316 : cluster [DBG] pgmap v294: 65 pgs: 65 active+clean; 3.4 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-20T09:01:07.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:06 smithi102 conmon[36472]: cluster 2021-11-20T09:01:05.755740+0000 mon.smithi093 (mon.0) 869 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:standby 2021-11-20T09:01:07.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:06 smithi102 conmon[36472]: cluster 2021-11-20T09:01:05.755824+0000 mon.smithi093 (mon.0) 870 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:07.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:06 smithi102 conmon[36472]: audit 2021-11-20T09:01:06.118560+0000 mon.smithi093 (mon. 2021-11-20T09:01:07.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:06 smithi102 conmon[36472]: 0) 871 : audit [DBG] from='client.? 172.21.15.93:0/2667712843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:07.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 -- 172.21.15.93:0/1321726264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 msgr2=0x7f972c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:07.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 --2- 172.21.15.93:0/1321726264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c0f6c80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9714004660 tx=0x7f9714009b30).stop 2021-11-20T09:01:07.266 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 -- 172.21.15.93:0/1321726264 shutdown_connections 2021-11-20T09:01:07.267 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 --2- 172.21.15.93:0/1321726264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c0f6c80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:07.268 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 --2- 172.21.15.93:0/1321726264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f972c0f5580 0x7f972c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:07.268 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 -- 172.21.15.93:0/1321726264 >> 172.21.15.93:0/1321726264 conn(0x7f972c0f0ce0 msgr2=0x7f972c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:07.268 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 -- 172.21.15.93:0/1321726264 shutdown_connections 2021-11-20T09:01:07.268 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 -- 172.21.15.93:0/1321726264 wait complete. 2021-11-20T09:01:07.269 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 Processor -- start 2021-11-20T09:01:07.269 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.260+0000 7f9732cbc700 1 -- start start 2021-11-20T09:01:07.269 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f9732cbc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f972c0f5580 0x7f972c060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:07.269 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f9732cbc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c10bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:07.269 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f9732cbc700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c10c1a0 con 0x7f972c0f6820 2021-11-20T09:01:07.270 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f9732cbc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c10c2e0 con 0x7f972c0f5580 2021-11-20T09:01:07.270 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f972bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c10bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:07.270 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f972bfff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c10bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44612/0 (socket says 172.21.15.93:44612) 2021-11-20T09:01:07.271 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f972bfff700 1 -- 172.21.15.93:0/4262503877 learned_addr learned my addr 172.21.15.93:0/4262503877 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:07.271 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f9730a58700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f972c0f5580 0x7f972c060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:07.271 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f972bfff700 1 -- 172.21.15.93:0/4262503877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f972c0f5580 msgr2=0x7f972c060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:07.271 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f972bfff700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f972c0f5580 0x7f972c060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:07.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f972bfff700 1 -- 172.21.15.93:0/4262503877 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714005040 con 0x7f972c0f6820 2021-11-20T09:01:07.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f972bfff700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c10bcb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f9714007a60 tx=0x7f9714009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:07.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.261+0000 7f97297fa700 1 -- 172.21.15.93:0/4262503877 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9714018070 con 0x7f972c0f6820 2021-11-20T09:01:07.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.262+0000 7f97297fa700 1 -- 172.21.15.93:0/4262503877 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9714013410 con 0x7f972c0f6820 2021-11-20T09:01:07.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.262+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c10c4e0 con 0x7f972c0f6820 2021-11-20T09:01:07.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.262+0000 7f97297fa700 1 -- 172.21.15.93:0/4262503877 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f971401c3e0 con 0x7f972c0f6820 2021-11-20T09:01:07.274 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.262+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c10ca40 con 0x7f972c0f6820 2021-11-20T09:01:07.274 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.262+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c027a00 con 0x7f972c0f6820 2021-11-20T09:01:07.274 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.263+0000 7f97297fa700 1 -- 172.21.15.93:0/4262503877 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f971401d420 con 0x7f972c0f6820 2021-11-20T09:01:07.275 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.263+0000 7f97297fa700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f971805c2a0 0x7f971805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:07.275 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.263+0000 7f97297fa700 1 -- 172.21.15.93:0/4262503877 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f97140781d0 con 0x7f972c0f6820 2021-11-20T09:01:07.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.263+0000 7f9730a58700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f971805c2a0 0x7f971805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:07.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.264+0000 7f9730a58700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f971805c2a0 0x7f971805e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f972c0f6680 tx=0x7f971c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:07.277 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.267+0000 7f97297fa700 1 -- 172.21.15.93:0/4262503877 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f971404a990 con 0x7f972c0f6820 2021-11-20T09:01:07.452 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.451+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f972c0fdd10 con 0x7f972c0f6820 2021-11-20T09:01:07.453 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.452+0000 7f97297fa700 1 -- 172.21.15.93:0/4262503877 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f971404dfb0 con 0x7f972c0f6820 2021-11-20T09:01:07.454 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:07.454 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:07.457 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f971805c2a0 msgr2=0x7f971805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:07.458 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f971805c2a0 0x7f971805e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f972c0f6680 tx=0x7f971c009040).stop 2021-11-20T09:01:07.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 msgr2=0x7f972c10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:07.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c10bcb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f9714007a60 tx=0x7f9714009b30).stop 2021-11-20T09:01:07.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 shutdown_connections 2021-11-20T09:01:07.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f971805c2a0 0x7f971805e760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:07.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f972c0f5580 0x7f972c060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:07.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 --2- 172.21.15.93:0/4262503877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0f6820 0x7f972c10bcb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:07.461 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 >> 172.21.15.93:0/4262503877 conn(0x7f972c0f0ce0 msgr2=0x7f972c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:07.461 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 shutdown_connections 2021-11-20T09:01:07.461 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:07.454+0000 7f9732cbc700 1 -- 172.21.15.93:0/4262503877 wait complete. 2021-11-20T09:01:07.462 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:07.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:07.466 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:08.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:07 smithi102 conmon[36472]: audit 2021-11-20T09:01:07.454057+0000 mon.smithi093 (mon.0 2021-11-20T09:01:08.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:07 smithi102 conmon[36472]: ) 872 : audit [DBG] from='client.? 172.21.15.93:0/4262503877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:08.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:07 smithi093 conmon[33522]: audit 2021-11-20T09:01:07.454057+0000 mon.smithi093 (mon.0) 872 : audit [DBG] from='client.? 172.21.15.93:0/4262503877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:08.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:07 smithi093 conmon[33522]: 2021-11-20T09:01:08.467 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:08.596 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 -- 172.21.15.93:0/2724371021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 msgr2=0x7f3eb00f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:08.597 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/2724371021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb00f6a90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f3eac004660 tx=0x7f3eac009b30).stop 2021-11-20T09:01:08.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 -- 172.21.15.93:0/2724371021 shutdown_connections 2021-11-20T09:01:08.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/2724371021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb00f7910 0x7f3eb00f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:08.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/2724371021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb00f6a90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:08.598 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 -- 172.21.15.93:0/2724371021 >> 172.21.15.93:0/2724371021 conn(0x7f3eb00f1db0 msgr2=0x7f3eb00f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:08.599 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 -- 172.21.15.93:0/2724371021 shutdown_connections 2021-11-20T09:01:08.599 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.593+0000 7f3eb7efc700 1 -- 172.21.15.93:0/2724371021 wait complete. 2021-11-20T09:01:08.599 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb7efc700 1 Processor -- start 2021-11-20T09:01:08.599 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb7efc700 1 -- start start 2021-11-20T09:01:08.599 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb7efc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:08.600 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb7efc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb00f7910 0x7f3eb0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:08.600 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb7efc700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb006a310 con 0x7f3eb00f6670 2021-11-20T09:01:08.600 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb7efc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb006a450 con 0x7f3eb00f7910 2021-11-20T09:01:08.601 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5c98700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:08.601 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5c98700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44620/0 (socket says 172.21.15.93:44620) 2021-11-20T09:01:08.601 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5c98700 1 -- 172.21.15.93:0/3795541639 learned_addr learned my addr 172.21.15.93:0/3795541639 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:08.601 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5497700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb00f7910 0x7f3eb0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:08.602 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5c98700 1 -- 172.21.15.93:0/3795541639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb00f7910 msgr2=0x7f3eb0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:08.602 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5c98700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb00f7910 0x7f3eb0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:08.602 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5c98700 1 -- 172.21.15.93:0/3795541639 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac005040 con 0x7f3eb00f6670 2021-11-20T09:01:08.602 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.594+0000 7f3eb5c98700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb006b7c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f3eac009f50 tx=0x7f3eac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:08.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.595+0000 7f3ea67fc700 1 -- 172.21.15.93:0/3795541639 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eac018070 con 0x7f3eb00f6670 2021-11-20T09:01:08.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.595+0000 7f3ea67fc700 1 -- 172.21.15.93:0/3795541639 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3eac013410 con 0x7f3eb00f6670 2021-11-20T09:01:08.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.595+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb006a6d0 con 0x7f3eb00f6670 2021-11-20T09:01:08.603 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.595+0000 7f3ea67fc700 1 -- 172.21.15.93:0/3795541639 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eac01c620 con 0x7f3eb00f6670 2021-11-20T09:01:08.604 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.595+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb006ac30 con 0x7f3eb00f6670 2021-11-20T09:01:08.604 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.596+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb0027a00 con 0x7f3eb00f6670 2021-11-20T09:01:08.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.596+0000 7f3ea67fc700 1 -- 172.21.15.93:0/3795541639 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3eac01d420 con 0x7f3eb00f6670 2021-11-20T09:01:08.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.596+0000 7f3ea67fc700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3e9c05c2a0 0x7f3e9c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:08.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.597+0000 7f3ea67fc700 1 -- 172.21.15.93:0/3795541639 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f3eac078320 con 0x7f3eb00f6670 2021-11-20T09:01:08.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.597+0000 7f3eb5497700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3e9c05c2a0 0x7f3e9c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:08.605 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.597+0000 7f3eb5497700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3e9c05c2a0 0x7f3e9c05e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f3ea000b790 tx=0x7f3ea0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:08.606 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.600+0000 7f3ea67fc700 1 -- 172.21.15.93:0/3795541639 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3eac04aa40 con 0x7f3eb00f6670 2021-11-20T09:01:08.786 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.785+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eb00fee80 con 0x7f3eb00f6670 2021-11-20T09:01:08.787 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.786+0000 7f3ea67fc700 1 -- 172.21.15.93:0/3795541639 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f3eac04e060 con 0x7f3eb00f6670 2021-11-20T09:01:08.787 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:08.788 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:08.789 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.787+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3e9c05c2a0 msgr2=0x7f3e9c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:08.789 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.787+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3e9c05c2a0 0x7f3e9c05e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f3ea000b790 tx=0x7f3ea0008040).stop 2021-11-20T09:01:08.790 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.787+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 msgr2=0x7f3eb006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:08.790 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.787+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb006b7c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f3eac009f50 tx=0x7f3eac009b30).stop 2021-11-20T09:01:08.790 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.788+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 shutdown_connections 2021-11-20T09:01:08.791 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.788+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f3e9c05c2a0 0x7f3e9c05e760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:08.791 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.788+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00f6670 0x7f3eb006b7c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:08.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.788+0000 7f3eb7efc700 1 --2- 172.21.15.93:0/3795541639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb00f7910 0x7f3eb0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:08.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.788+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 >> 172.21.15.93:0/3795541639 conn(0x7f3eb00f1db0 msgr2=0x7f3eb00fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:08.792 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.788+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 shutdown_connections 2021-11-20T09:01:08.793 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:08.788+0000 7f3eb7efc700 1 -- 172.21.15.93:0/3795541639 wait complete. 2021-11-20T09:01:08.793 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:08.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:08.800 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:09.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:08 smithi102 conmon[36472]: cluster 2021-11-20T09:01:07.585658+0000 mgr.smithi093.qdikkm (mgr 2021-11-20T09:01:09.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:08 smithi102 conmon[36472]: .14162) 317 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:09.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:08 smithi093 conmon[33522]: cluster 2021-11-20T09:01:07.585658+0000 mgr.smithi093.qdikkm (mgr.14162) 317 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:09.801 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:09.934 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 -- 172.21.15.93:0/3032895436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f7940 msgr2=0x7ff3e00f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:09.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/3032895436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f7940 0x7ff3e00f7da0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7ff3d4004660 tx=0x7ff3d4009b30).stop 2021-11-20T09:01:09.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 -- 172.21.15.93:0/3032895436 shutdown_connections 2021-11-20T09:01:09.935 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/3032895436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f7940 0x7ff3e00f7da0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:09.936 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/3032895436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e00f66a0 0x7ff3e00f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:09.936 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 -- 172.21.15.93:0/3032895436 >> 172.21.15.93:0/3032895436 conn(0x7ff3e00f1e00 msgr2=0x7ff3e00f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:09.937 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 -- 172.21.15.93:0/3032895436 shutdown_connections 2021-11-20T09:01:09.937 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.932+0000 7ff3e4cea700 1 -- 172.21.15.93:0/3032895436 wait complete. 2021-11-20T09:01:09.938 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3e4cea700 1 Processor -- start 2021-11-20T09:01:09.938 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3e4cea700 1 -- start start 2021-11-20T09:01:09.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3e4cea700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f66a0 0x7ff3e006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:09.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3e4cea700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e00f7940 0x7ff3e0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:09.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3e4cea700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e006bd70 con 0x7ff3e00f66a0 2021-11-20T09:01:09.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3e4cea700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e006a310 con 0x7ff3e00f7940 2021-11-20T09:01:09.939 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3de59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f66a0 0x7ff3e006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:09.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3de59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f66a0 0x7ff3e006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44632/0 (socket says 172.21.15.93:44632) 2021-11-20T09:01:09.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3de59c700 1 -- 172.21.15.93:0/1009369088 learned_addr learned my addr 172.21.15.93:0/1009369088 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:09.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3ddd9b700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e00f7940 0x7ff3e0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:09.940 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3de59c700 1 -- 172.21.15.93:0/1009369088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e00f7940 msgr2=0x7ff3e0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:09.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3de59c700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e00f7940 0x7ff3e0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:09.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3de59c700 1 -- 172.21.15.93:0/1009369088 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d4005040 con 0x7ff3e00f66a0 2021-11-20T09:01:09.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.933+0000 7ff3ddd9b700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e00f7940 0x7ff3e0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:01:09.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.934+0000 7ff3de59c700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f66a0 0x7ff3e006b7c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7ff3d000eb70 tx=0x7ff3d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:09.941 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.934+0000 7ff3ceffd700 1 -- 172.21.15.93:0/1009369088 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d00092e0 con 0x7ff3e00f66a0 2021-11-20T09:01:09.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.934+0000 7ff3ceffd700 1 -- 172.21.15.93:0/1009369088 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3d000f040 con 0x7ff3e00f66a0 2021-11-20T09:01:09.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.934+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e006a590 con 0x7ff3e00f66a0 2021-11-20T09:01:09.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.934+0000 7ff3ceffd700 1 -- 172.21.15.93:0/1009369088 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d000c9a0 con 0x7ff3e00f66a0 2021-11-20T09:01:09.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.934+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e006ab50 con 0x7ff3e00f66a0 2021-11-20T09:01:09.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.935+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e0027a00 con 0x7ff3e00f66a0 2021-11-20T09:01:09.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.935+0000 7ff3ceffd700 1 -- 172.21.15.93:0/1009369088 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff3d001b030 con 0x7ff3e00f66a0 2021-11-20T09:01:09.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.935+0000 7ff3ceffd700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3c805c2a0 0x7ff3c805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:09.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.936+0000 7ff3ceffd700 1 -- 172.21.15.93:0/1009369088 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7ff3d0076480 con 0x7ff3e00f66a0 2021-11-20T09:01:09.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.936+0000 7ff3ddd9b700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3c805c2a0 0x7ff3c805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:09.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.936+0000 7ff3ddd9b700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3c805c2a0 0x7ff3c805e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff3d4000c00 tx=0x7ff3d400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:09.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:09.939+0000 7ff3ceffd700 1 -- 172.21.15.93:0/1009369088 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3d0048cc0 con 0x7ff3e00f66a0 2021-11-20T09:01:09.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:09 smithi093 conmon[33522]: audit 2021-11-20T09:01:08.787916+0000 mon.smithi093 (mon.0) 873 : audit [DBG] from='client.? 172.21.15.93:0/3795541639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:09.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:09 smithi093 conmon[33522]: 2021-11-20T09:01:10.125 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.126+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e00fee30 con 0x7ff3e00f66a0 2021-11-20T09:01:10.126 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.127+0000 7ff3ceffd700 1 -- 172.21.15.93:0/1009369088 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7ff3d0019090 con 0x7ff3e00f66a0 2021-11-20T09:01:10.126 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:10.126 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:10.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3c805c2a0 msgr2=0x7ff3c805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:10.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3c805c2a0 0x7ff3c805e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff3d4000c00 tx=0x7ff3d400b040).stop 2021-11-20T09:01:10.128 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f66a0 msgr2=0x7ff3e006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:10.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f66a0 0x7ff3e006b7c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7ff3d000eb70 tx=0x7ff3d000b040).stop 2021-11-20T09:01:10.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 shutdown_connections 2021-11-20T09:01:10.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3c805c2a0 0x7ff3c805e760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:10.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e00f66a0 0x7ff3e006b7c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:10.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 --2- 172.21.15.93:0/1009369088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e00f7940 0x7ff3e0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:10.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 >> 172.21.15.93:0/1009369088 conn(0x7ff3e00f1e00 msgr2=0x7ff3e00fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:10.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 shutdown_connections 2021-11-20T09:01:10.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:10.128+0000 7ff3e4cea700 1 -- 172.21.15.93:0/1009369088 wait complete. 2021-11-20T09:01:10.131 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:10.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:10.140 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:10.179 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:09 smithi102 conmon[36472]: audit 2021-11-20T09:01:08.787916+0000 mon.smithi093 (mon.0) 873 : audit [DBG] 2021-11-20T09:01:10.180 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:09 smithi102 conmon[36472]: from='client.? 172.21.15.93:0/3795541639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:11.141 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:11.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:10 smithi102 conmon[36472]: cluster 2021-11-20T09:01:09.586173+0000 mgr.smithi093.qdikkm (mgr.14162) 318 : cluster [DBG] pgmap v296: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:11.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:10 smithi102 conmon[36472]: audit 2021-11-20T09:01:10.127777+0000 mon.smithi093 (mon.0) 874 : audit [DBG] from='client.? 172.21.15.93:0/1009369088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:11.169 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:10 smithi093 conmon[33522]: cluster 2021-11-20T09:01:09.586173+0000 mgr.smithi093.qdikkm (mgr.14162) 318 : cluster [DBG] 2021-11-20T09:01:11.169 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:10 smithi093 conmon[33522]: pgmap v296: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:11.170 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:10 smithi093 conmon[33522]: audit 2021-11-20T09:01:10.127777+0000 mon.smithi093 (mon.0) 874 : audit [DBG] from='client.? 172.21.15.93:0/1009369088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:11.271 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 -- 172.21.15.93:0/3611799659 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 msgr2=0x7f18c80f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:11.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 --2- 172.21.15.93:0/3611799659 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c80f5880 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f18c40099e0 tx=0x7f18c4004d10).stop 2021-11-20T09:01:11.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 -- 172.21.15.93:0/3611799659 shutdown_connections 2021-11-20T09:01:11.272 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 --2- 172.21.15.93:0/3611799659 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c80f6700 0x7f18c80f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:11.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 --2- 172.21.15.93:0/3611799659 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c80f5880 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:11.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 -- 172.21.15.93:0/3611799659 >> 172.21.15.93:0/3611799659 conn(0x7f18c80f0ba0 msgr2=0x7f18c80f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:11.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 -- 172.21.15.93:0/3611799659 shutdown_connections 2021-11-20T09:01:11.273 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.267+0000 7f18ce07c700 1 -- 172.21.15.93:0/3611799659 wait complete. 2021-11-20T09:01:11.274 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18ce07c700 1 Processor -- start 2021-11-20T09:01:11.274 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18ce07c700 1 -- start start 2021-11-20T09:01:11.274 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18ce07c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c810bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:11.275 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18ce07c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c80f6700 0x7f18c810e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:11.275 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18ce07c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c80600e0 con 0x7f18c80f5460 2021-11-20T09:01:11.275 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18ce07c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c8060220 con 0x7f18c80f6700 2021-11-20T09:01:11.275 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18cd07a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c810bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:11.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18cd07a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c810bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44642/0 (socket says 172.21.15.93:44642) 2021-11-20T09:01:11.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18cd07a700 1 -- 172.21.15.93:0/2094531588 learned_addr learned my addr 172.21.15.93:0/2094531588 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:11.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18cc879700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c80f6700 0x7f18c810e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:11.276 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18cd07a700 1 -- 172.21.15.93:0/2094531588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c80f6700 msgr2=0x7f18c810e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:11.277 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18cd07a700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c80f6700 0x7f18c810e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:11.277 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.268+0000 7f18cd07a700 1 -- 172.21.15.93:0/2094531588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c4009640 con 0x7f18c80f5460 2021-11-20T09:01:11.277 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.269+0000 7f18cd07a700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c810bf70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f18c4007830 tx=0x7f18c4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:11.277 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.269+0000 7f18bdffb700 1 -- 172.21.15.93:0/2094531588 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18c400e030 con 0x7f18c80f5460 2021-11-20T09:01:11.278 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.269+0000 7f18bdffb700 1 -- 172.21.15.93:0/2094531588 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18c401d8d0 con 0x7f18c80f5460 2021-11-20T09:01:11.278 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.269+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c810e960 con 0x7f18c80f5460 2021-11-20T09:01:11.278 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.269+0000 7f18bdffb700 1 -- 172.21.15.93:0/2094531588 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18c4018070 con 0x7f18c80f5460 2021-11-20T09:01:11.278 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.269+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c810ecb0 con 0x7f18c80f5460 2021-11-20T09:01:11.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.270+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c80f4850 con 0x7f18c80f5460 2021-11-20T09:01:11.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.270+0000 7f18bdffb700 1 -- 172.21.15.93:0/2094531588 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f18c401da40 con 0x7f18c80f5460 2021-11-20T09:01:11.279 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.270+0000 7f18bdffb700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f18b405c300 0x7f18b405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:11.280 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.271+0000 7f18bdffb700 1 -- 172.21.15.93:0/2094531588 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f18c40782f0 con 0x7f18c80f5460 2021-11-20T09:01:11.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.271+0000 7f18cc879700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f18b405c300 0x7f18b405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:11.281 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.271+0000 7f18cc879700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f18b405c300 0x7f18b405e7c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f18b80041d0 tx=0x7f18b8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:11.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.274+0000 7f18bdffb700 1 -- 172.21.15.93:0/2094531588 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18c404aab0 con 0x7f18c80f5460 2021-11-20T09:01:11.455 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.456+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18c8045ad0 con 0x7f18c80f5460 2021-11-20T09:01:11.455 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.457+0000 7f18bdffb700 1 -- 172.21.15.93:0/2094531588 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f18c404e0d0 con 0x7f18c80f5460 2021-11-20T09:01:11.456 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:11.456 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:11.457 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f18b405c300 msgr2=0x7f18b405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:11.458 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f18b405c300 0x7f18b405e7c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f18b80041d0 tx=0x7f18b8009040).stop 2021-11-20T09:01:11.458 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 msgr2=0x7f18c810bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:11.458 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c810bf70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f18c4007830 tx=0x7f18c4004d10).stop 2021-11-20T09:01:11.458 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 shutdown_connections 2021-11-20T09:01:11.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f18b405c300 0x7f18b405e7c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:11.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18c80f5460 0x7f18c810bf70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:11.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 --2- 172.21.15.93:0/2094531588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18c80f6700 0x7f18c810e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:11.459 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 >> 172.21.15.93:0/2094531588 conn(0x7f18c80f0ba0 msgr2=0x7f18c80fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:11.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 shutdown_connections 2021-11-20T09:01:11.460 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:11.458+0000 7f18ce07c700 1 -- 172.21.15.93:0/2094531588 wait complete. 2021-11-20T09:01:11.460 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:11.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:11.469 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:12.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:11 smithi102 conmon[36472]: audit 2021-11-20T09:01:11.457585+0000 mon.smithi093 (mon.0) 875 : audit [DBG] 2021-11-20T09:01:12.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:11 smithi102 conmon[36472]: from='client.? 172.21.15.93:0/2094531588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:12.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:11 smithi093 conmon[33522]: audit 2021-11-20T09: 2021-11-20T09:01:12.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:11 smithi093 conmon[33522]: 01:11.457585+0000 mon.smithi093 (mon.0) 875 : audit [DBG] from='client.? 172.21.15.93:0/2094531588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:12.474 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:12.610 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2124351356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f4a40 msgr2=0x7fcfe00f4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:12.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2124351356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f4a40 0x7fcfe00f4e60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fcfdc004660 tx=0x7fcfdc009b30).stop 2021-11-20T09:01:12.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2124351356 shutdown_connections 2021-11-20T09:01:12.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2124351356 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfe00f53e0 0x7fcfe00fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:12.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2124351356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f4a40 0x7fcfe00f4e60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:12.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2124351356 >> 172.21.15.93:0/2124351356 conn(0x7fcfe00f0440 msgr2=0x7fcfe00f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:12.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2124351356 shutdown_connections 2021-11-20T09:01:12.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2124351356 wait complete. 2021-11-20T09:01:12.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 Processor -- start 2021-11-20T09:01:12.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- start start 2021-11-20T09:01:12.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfe00f4a40 0x7fcfe0106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:12.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f53e0 0x7fcfe0107370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:12.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0107920 con 0x7fcfe00f53e0 2021-11-20T09:01:12.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.608+0000 7fcfe7f19700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0107a60 con 0x7fcfe00f4a40 2021-11-20T09:01:12.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe54b4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f53e0 0x7fcfe0107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:12.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe54b4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f53e0 0x7fcfe0107370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44654/0 (socket says 172.21.15.93:44654) 2021-11-20T09:01:12.614 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe54b4700 1 -- 172.21.15.93:0/2780186221 learned_addr learned my addr 172.21.15.93:0/2780186221 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:12.615 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe5cb5700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfe00f4a40 0x7fcfe0106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:12.615 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe54b4700 1 -- 172.21.15.93:0/2780186221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfe00f4a40 msgr2=0x7fcfe0106e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:12.615 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe54b4700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfe00f4a40 0x7fcfe0106e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:12.616 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe54b4700 1 -- 172.21.15.93:0/2780186221 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfdc005040 con 0x7fcfe00f53e0 2021-11-20T09:01:12.616 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe54b4700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f53e0 0x7fcfe0107370 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fcfd000c790 tx=0x7fcfd0007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:12.617 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfd67fc700 1 -- 172.21.15.93:0/2780186221 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd000e680 con 0x7fcfe00f53e0 2021-11-20T09:01:12.617 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfd67fc700 1 -- 172.21.15.93:0/2780186221 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfd000ec90 con 0x7fcfe00f53e0 2021-11-20T09:01:12.617 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe010c500 con 0x7fcfe00f53e0 2021-11-20T09:01:12.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfd67fc700 1 -- 172.21.15.93:0/2780186221 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd0013d60 con 0x7fcfe00f53e0 2021-11-20T09:01:12.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.609+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe010cac0 con 0x7fcfe00f53e0 2021-11-20T09:01:12.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.611+0000 7fcfd67fc700 1 -- 172.21.15.93:0/2780186221 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcfd001a060 con 0x7fcfe00f53e0 2021-11-20T09:01:12.618 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.611+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe0027a00 con 0x7fcfe00f53e0 2021-11-20T09:01:12.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.611+0000 7fcfd67fc700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcfcc05c2a0 0x7fcfcc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:12.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.611+0000 7fcfd67fc700 1 -- 172.21.15.93:0/2780186221 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcfd00765f0 con 0x7fcfe00f53e0 2021-11-20T09:01:12.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.611+0000 7fcfe5cb5700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcfcc05c2a0 0x7fcfcc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:12.619 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.611+0000 7fcfe5cb5700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcfcc05c2a0 0x7fcfcc05e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fcfdc004e30 tx=0x7fcfdc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:12.620 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.615+0000 7fcfd67fc700 1 -- 172.21.15.93:0/2780186221 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfd004c7e0 con 0x7fcfe00f53e0 2021-11-20T09:01:12.799 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.800+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfe00fee30 con 0x7fcfe00f53e0 2021-11-20T09:01:12.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.801+0000 7fcfd67fc700 1 -- 172.21.15.93:0/2780186221 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7fcfd004c3d0 con 0x7fcfe00f53e0 2021-11-20T09:01:12.800 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:12.801 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:12.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.802+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcfcc05c2a0 msgr2=0x7fcfcc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:12.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.802+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcfcc05c2a0 0x7fcfcc05e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fcfdc004e30 tx=0x7fcfdc009b30).stop 2021-11-20T09:01:12.802 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.802+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f53e0 msgr2=0x7fcfe0107370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:12.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.802+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f53e0 0x7fcfe0107370 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fcfd000c790 tx=0x7fcfd0007d70).stop 2021-11-20T09:01:12.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.803+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 shutdown_connections 2021-11-20T09:01:12.803 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.803+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fcfcc05c2a0 0x7fcfcc05e760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:12.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.803+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfe00f4a40 0x7fcfe0106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:12.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.803+0000 7fcfe7f19700 1 --2- 172.21.15.93:0/2780186221 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcfe00f53e0 0x7fcfe0107370 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:12.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.803+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 >> 172.21.15.93:0/2780186221 conn(0x7fcfe00f0440 msgr2=0x7fcfe00f84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:12.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.803+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 shutdown_connections 2021-11-20T09:01:12.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:12.803+0000 7fcfe7f19700 1 -- 172.21.15.93:0/2780186221 wait complete. 2021-11-20T09:01:12.806 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:12.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:12.814 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:13.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:12 smithi102 conmon[36472]: cluster 2021-11-20T09:01:11.586740+0000 mgr.smithi093.qdikkm (mgr.14162) 319 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 196 B/s wr, 1 op/s 2021-11-20T09:01:13.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:12 smithi093 conmon[33522]: cluster 2021-11-20T09:01:11.586740+0000 mgr.smithi093.qdikkm (mgr.14162) 319 2021-11-20T09:01:13.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:12 smithi093 conmon[33522]: : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 196 B/s wr, 1 op/s 2021-11-20T09:01:13.815 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:13.942 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 -- 172.21.15.93:0/256558054 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 msgr2=0x7f8d400f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:13.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/256558054 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f7a80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f8d28004660 tx=0x7f8d28009b30).stop 2021-11-20T09:01:13.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 -- 172.21.15.93:0/256558054 shutdown_connections 2021-11-20T09:01:13.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/256558054 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f7a80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:13.943 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/256558054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d400f4960 0x7f8d400f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:13.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 -- 172.21.15.93:0/256558054 >> 172.21.15.93:0/256558054 conn(0x7f8d400f0440 msgr2=0x7f8d400f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:13.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 -- 172.21.15.93:0/256558054 shutdown_connections 2021-11-20T09:01:13.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 -- 172.21.15.93:0/256558054 wait complete. 2021-11-20T09:01:13.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.941+0000 7f8d45f2c700 1 Processor -- start 2021-11-20T09:01:13.944 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d45f2c700 1 -- start start 2021-11-20T09:01:13.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d45f2c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d400f4960 0x7f8d400f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:13.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d45f2c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:13.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d45f2c700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400f5e80 con 0x7f8d400f5300 2021-11-20T09:01:13.945 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d45f2c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400f5fc0 con 0x7f8d400f4960 2021-11-20T09:01:13.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d36ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:13.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d36ffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44662/0 (socket says 172.21.15.93:44662) 2021-11-20T09:01:13.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d36ffd700 1 -- 172.21.15.93:0/641658269 learned_addr learned my addr 172.21.15.93:0/641658269 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:13.946 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d3f7fe700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d400f4960 0x7f8d400f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:13.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d36ffd700 1 -- 172.21.15.93:0/641658269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d400f4960 msgr2=0x7f8d400f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:13.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d36ffd700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d400f4960 0x7f8d400f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:13.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d36ffd700 1 -- 172.21.15.93:0/641658269 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d28005040 con 0x7f8d400f5300 2021-11-20T09:01:13.947 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d3f7fe700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d400f4960 0x7f8d400f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:01:13.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d36ffd700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f5990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f8d28009f50 tx=0x7f8d28009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:13.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d3cff9700 1 -- 172.21.15.93:0/641658269 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d28018070 con 0x7f8d400f5300 2021-11-20T09:01:13.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d400f6240 con 0x7f8d400f5300 2021-11-20T09:01:13.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d3cff9700 1 -- 172.21.15.93:0/641658269 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d28013410 con 0x7f8d400f5300 2021-11-20T09:01:13.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.942+0000 7f8d3cff9700 1 -- 172.21.15.93:0/641658269 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d2801c620 con 0x7f8d400f5300 2021-11-20T09:01:13.949 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.943+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d400f67a0 con 0x7f8d400f5300 2021-11-20T09:01:13.949 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.943+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d40027a00 con 0x7f8d400f5300 2021-11-20T09:01:13.949 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.944+0000 7f8d3cff9700 1 -- 172.21.15.93:0/641658269 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8d2801d420 con 0x7f8d400f5300 2021-11-20T09:01:13.951 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.944+0000 7f8d3cff9700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8d2c05c2a0 0x7f8d2c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:13.951 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.944+0000 7f8d3cff9700 1 -- 172.21.15.93:0/641658269 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8d28078280 con 0x7f8d400f5300 2021-11-20T09:01:13.952 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.944+0000 7f8d3f7fe700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8d2c05c2a0 0x7f8d2c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:13.952 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.945+0000 7f8d3f7fe700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8d2c05c2a0 0x7f8d2c05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8d4005f7e0 tx=0x7f8d30008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:13.952 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:13.948+0000 7f8d3cff9700 1 -- 172.21.15.93:0/641658269 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d2804aa40 con 0x7f8d400f5300 2021-11-20T09:01:14.129 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.130+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d400feef0 con 0x7f8d400f5300 2021-11-20T09:01:14.130 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.131+0000 7f8d3cff9700 1 -- 172.21.15.93:0/641658269 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f8d400feef0 con 0x7f8d400f5300 2021-11-20T09:01:14.130 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:14.131 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:14.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8d2c05c2a0 msgr2=0x7f8d2c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:14.133 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8d2c05c2a0 0x7f8d2c05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8d4005f7e0 tx=0x7f8d30008040).stop 2021-11-20T09:01:14.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 msgr2=0x7f8d400f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:14.134 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f5990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f8d28009f50 tx=0x7f8d28009b30).stop 2021-11-20T09:01:14.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 shutdown_connections 2021-11-20T09:01:14.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f8d2c05c2a0 0x7f8d2c05e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:14.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d400f4960 0x7f8d400f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:14.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 --2- 172.21.15.93:0/641658269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400f5300 0x7f8d400f5990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:14.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 >> 172.21.15.93:0/641658269 conn(0x7f8d400f0440 msgr2=0x7f8d400fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:14.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 shutdown_connections 2021-11-20T09:01:14.136 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:14.133+0000 7f8d45f2c700 1 -- 172.21.15.93:0/641658269 wait complete. 2021-11-20T09:01:14.137 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:14.142 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:13 smithi093 conmon[33522]: audit 2021-11-20T09:01:12.801885+0000 mon.smithi093 (mon.0) 876 : audit [DBG] from='client.? 172.21.15.93:0/2780186221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:14.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:14.144 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:14.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:13 smithi102 conmon[36472]: audit 2021-11-20T09:01:12.801885+0000 mon.smithi093 (mon.0) 876 : audit [DBG] from='client.? 172.21.15.93:0/2780186221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:14.981 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:14 smithi102 conmon[36472]: cluster 2021-11-20T09:01:13.587238+0000 mgr.smithi093.qdikkm (mgr.14162) 320 : 2021-11-20T09:01:14.981 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:14 smithi102 conmon[36472]: cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:01:14.981 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:14 smithi102 conmon[36472]: audit 2021-11-20T09:01:14.131987+0000 mon.smithi093 (mon.0) 877 : audit [DBG] from='client.? 172.21.15.93:0/641658269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:15.145 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:15.174 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:14 smithi093 conmon[33522]: cluster 2021-11-20T09:01:13.587238+0000 mgr.smithi093.qdikkm (mgr.14162) 320 : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:01:15.175 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:14 smithi093 conmon[33522]: audit 2021-11-20T09:01:14.131987+0000 mon.smithi093 (mon.0) 877 : audit [DBG] from='client.? 172.21.15.93:0/641658269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:15.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.278+0000 7f4066864700 1 -- 172.21.15.93:0/803040953 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40600f9790 msgr2=0x7f40600fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:15.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.278+0000 7f4066864700 1 --2- 172.21.15.93:0/803040953 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40600f9790 0x7f40600fbb80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f4050004660 tx=0x7f4050009b30).stop 2021-11-20T09:01:15.283 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.278+0000 7f4066864700 1 -- 172.21.15.93:0/803040953 shutdown_connections 2021-11-20T09:01:15.284 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.278+0000 7f4066864700 1 --2- 172.21.15.93:0/803040953 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40600f9790 0x7f40600fbb80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:15.284 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.278+0000 7f4066864700 1 --2- 172.21.15.93:0/803040953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4060060410 0x7f40600f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:15.284 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 -- 172.21.15.93:0/803040953 >> 172.21.15.93:0/803040953 conn(0x7f40600f0c30 msgr2=0x7f40600f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:15.284 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 -- 172.21.15.93:0/803040953 shutdown_connections 2021-11-20T09:01:15.286 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 -- 172.21.15.93:0/803040953 wait complete. 2021-11-20T09:01:15.286 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 Processor -- start 2021-11-20T09:01:15.287 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 -- start start 2021-11-20T09:01:15.287 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4060060410 0x7f40600f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:15.287 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40600f9790 0x7f40600f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:15.288 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40600f5670 con 0x7f4060060410 2021-11-20T09:01:15.288 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4066864700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40600f57b0 con 0x7f40600f9790 2021-11-20T09:01:15.289 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4065862700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4060060410 0x7f40600f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:15.289 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.279+0000 7f4065862700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4060060410 0x7f40600f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44672/0 (socket says 172.21.15.93:44672) 2021-11-20T09:01:15.289 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4065862700 1 -- 172.21.15.93:0/963999021 learned_addr learned my addr 172.21.15.93:0/963999021 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:15.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4065061700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40600f9790 0x7f40600f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:15.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4065862700 1 -- 172.21.15.93:0/963999021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40600f9790 msgr2=0x7f40600f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:15.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4065862700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40600f9790 0x7f40600f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:15.290 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4065862700 1 -- 172.21.15.93:0/963999021 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4050005040 con 0x7f4060060410 2021-11-20T09:01:15.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4065862700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4060060410 0x7f40600f6c90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f405c00e7f0 tx=0x7f405c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:15.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f40567fc700 1 -- 172.21.15.93:0/963999021 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405c0092e0 con 0x7f4060060410 2021-11-20T09:01:15.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f40567fc700 1 -- 172.21.15.93:0/963999021 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f405c00f040 con 0x7f4060060410 2021-11-20T09:01:15.291 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40600f5a90 con 0x7f4060060410 2021-11-20T09:01:15.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f40567fc700 1 -- 172.21.15.93:0/963999021 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405c014750 con 0x7f4060060410 2021-11-20T09:01:15.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.280+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406010ed20 con 0x7f4060060410 2021-11-20T09:01:15.292 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.281+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4060045ad0 con 0x7f4060060410 2021-11-20T09:01:15.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.281+0000 7f40567fc700 1 -- 172.21.15.93:0/963999021 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f405c01b030 con 0x7f4060060410 2021-11-20T09:01:15.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.282+0000 7f40567fc700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f404c05c300 0x7f404c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:15.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.282+0000 7f40567fc700 1 -- 172.21.15.93:0/963999021 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f405c0761c0 con 0x7f4060060410 2021-11-20T09:01:15.293 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.282+0000 7f4065061700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f404c05c300 0x7f404c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:15.294 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.282+0000 7f4065061700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f404c05c300 0x7f404c05e7c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f4050013530 tx=0x7f4050016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:15.294 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.286+0000 7f40567fc700 1 -- 172.21.15.93:0/963999021 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f405c0489d0 con 0x7f4060060410 2021-11-20T09:01:15.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.470+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4060027a00 con 0x7f4060060410 2021-11-20T09:01:15.470 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.471+0000 7f40567fc700 1 -- 172.21.15.93:0/963999021 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f405c019090 con 0x7f4060060410 2021-11-20T09:01:15.471 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:15.471 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:04.613266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24565},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24565":{"gid":24565,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":86,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6825/2583579411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2583579411},{"type":"v1","addr":"172.21.15.102:6825","nonce":2583579411}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:15.472 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f404c05c300 msgr2=0x7f404c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:15.473 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f404c05c300 0x7f404c05e7c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f4050013530 tx=0x7f4050016040).stop 2021-11-20T09:01:15.473 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4060060410 msgr2=0x7f40600f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:15.473 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4060060410 0x7f40600f6c90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f405c00e7f0 tx=0x7f405c00b040).stop 2021-11-20T09:01:15.473 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 shutdown_connections 2021-11-20T09:01:15.474 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f404c05c300 0x7f404c05e7c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:15.474 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4060060410 0x7f40600f6c90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:15.474 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 --2- 172.21.15.93:0/963999021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40600f9790 0x7f40600f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:15.474 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 >> 172.21.15.93:0/963999021 conn(0x7f40600f0c30 msgr2=0x7f40600f3020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:15.475 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 shutdown_connections 2021-11-20T09:01:15.475 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:15.473+0000 7f4066864700 1 -- 172.21.15.93:0/963999021 wait complete. 2021-11-20T09:01:15.475 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 90 2021-11-20T09:01:15.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:04.613266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24565}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 86, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6825/2583579411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 2583579411}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 2583579411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:15.484 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:clientreplay 2021-11-20T09:01:16.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:15 smithi102 conmon[36472]: cluster 2021-11-20T09:01:15.152161+0000 mon.smithi093 (mon.0) 878 : cluster 2021-11-20T09:01:16.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:15 smithi102 conmon[36472]: [INF] daemon mds.cephfs.smithi102.lhrlmh restarted 2021-11-20T09:01:16.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:15 smithi102 conmon[36472]: cluster 2021-11-20T09:01:15.160777+0000 mon.smithi093 (mon.0) 879 : cluster [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-20T09:01:16.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:15 smithi102 conmon[36472]: audit 2021-11-20T09:01:15.472014+0000 mon.smithi093 (mon.0) 880 : audit [DBG] 2021-11-20T09:01:16.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:15 smithi102 conmon[36472]: from='client.? 172.21.15.93:0/963999021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:16.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: debug 2021-11-20T09:01:15.786+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:01:16.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: cluster 2021-11-20T09:01 2021-11-20T09:01:16.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: :15.152161+0000 mon.smithi093 (mon.0) 878 : cluster [INF] daemon mds.cephfs.smithi102.lhrlmh restarted 2021-11-20T09:01:16.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: cluster 2021-11-20T09:01:15.160777+0000 mon.smithi093 (mon.0) 879 : cluster 2021-11-20T09:01:16.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-20T09:01:16.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: audit 2021-11 2021-11-20T09:01:16.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: -20T09:01:15. 2021-11-20T09:01:16.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:15 smithi093 conmon[33522]: 472014+0000 mon.smithi093 (mon.0) 880 : audit [DBG] from='client.? 172.21.15.93:0/963999021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:16.486 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:16.606 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.605+0000 7fb835316700 1 -- 172.21.15.93:0/1902043268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 msgr2=0x7fb83006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:16.606 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.605+0000 7fb835316700 1 --2- 172.21.15.93:0/1902043268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb83006a7d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fb818004660 tx=0x7fb818009b30).stop 2021-11-20T09:01:16.607 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.605+0000 7fb835316700 1 -- 172.21.15.93:0/1902043268 shutdown_connections 2021-11-20T09:01:16.607 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.605+0000 7fb835316700 1 --2- 172.21.15.93:0/1902043268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb83006a7d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:16.607 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.605+0000 7fb835316700 1 --2- 172.21.15.93:0/1902043268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83006b960 0x7fb830069e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:16.607 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.605+0000 7fb835316700 1 -- 172.21.15.93:0/1902043268 >> 172.21.15.93:0/1902043268 conn(0x7fb8300f1de0 msgr2=0x7fb8300f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:16.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 -- 172.21.15.93:0/1902043268 shutdown_connections 2021-11-20T09:01:16.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 -- 172.21.15.93:0/1902043268 wait complete. 2021-11-20T09:01:16.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 Processor -- start 2021-11-20T09:01:16.608 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 -- start start 2021-11-20T09:01:16.609 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb830109070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:16.609 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83006b960 0x7fb830109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:16.609 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb830109b10 con 0x7fb83006a350 2021-11-20T09:01:16.610 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb835316700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb830109c50 con 0x7fb83006b960 2021-11-20T09:01:16.610 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb82effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb830109070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:16.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb82effd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb830109070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44682/0 (socket says 172.21.15.93:44682) 2021-11-20T09:01:16.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.606+0000 7fb82effd700 1 -- 172.21.15.93:0/3940253018 learned_addr learned my addr 172.21.15.93:0/3940253018 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:16.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb82e7fc700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83006b960 0x7fb830109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:16.611 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb82effd700 1 -- 172.21.15.93:0/3940253018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83006b960 msgr2=0x7fb830109560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:16.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb82effd700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83006b960 0x7fb830109560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:16.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb82effd700 1 -- 172.21.15.93:0/3940253018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818005040 con 0x7fb83006a350 2021-11-20T09:01:16.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb82effd700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb830109070 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fb82000c760 tx=0x7fb820004d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:16.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb8277fe700 1 -- 172.21.15.93:0/3940253018 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb820004200 con 0x7fb83006a350 2021-11-20T09:01:16.612 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83010e6f0 con 0x7fb83006a350 2021-11-20T09:01:16.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb8277fe700 1 -- 172.21.15.93:0/3940253018 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8200109c0 con 0x7fb83006a350 2021-11-20T09:01:16.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb8277fe700 1 -- 172.21.15.93:0/3940253018 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8200038d0 con 0x7fb83006a350 2021-11-20T09:01:16.613 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.607+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83010ecb0 con 0x7fb83006a350 2021-11-20T09:01:16.615 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.608+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb830027a00 con 0x7fb83006a350 2021-11-20T09:01:16.615 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.609+0000 7fb8277fe700 1 -- 172.21.15.93:0/3940253018 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb820018020 con 0x7fb83006a350 2021-11-20T09:01:16.615 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.609+0000 7fb8277fe700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb81c05c2a0 0x7fb81c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:16.615 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.609+0000 7fb8277fe700 1 -- 172.21.15.93:0/3940253018 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fb820076210 con 0x7fb83006a350 2021-11-20T09:01:16.616 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.609+0000 7fb82e7fc700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb81c05c2a0 0x7fb81c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:16.616 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.609+0000 7fb82e7fc700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb81c05c2a0 0x7fb81c05e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fb818013580 tx=0x7fb818009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:16.616 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.614+0000 7fb8277fe700 1 -- 172.21.15.93:0/3940253018 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8200462b0 con 0x7fb83006a350 2021-11-20T09:01:16.800 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.801+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb830101000 con 0x7fb83006a350 2021-11-20T09:01:16.801 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.803+0000 7fb8277fe700 1 -- 172.21.15.93:0/3940253018 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2189 (secure 0 0 0) 0x7fb820048970 con 0x7fb83006a350 2021-11-20T09:01:16.802 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:16.802 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:16.795621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:16.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb81c05c2a0 msgr2=0x7fb81c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:16.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb81c05c2a0 0x7fb81c05e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fb818013580 tx=0x7fb818009b30).stop 2021-11-20T09:01:16.804 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 msgr2=0x7fb830109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:16.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb830109070 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fb82000c760 tx=0x7fb820004d20).stop 2021-11-20T09:01:16.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 shutdown_connections 2021-11-20T09:01:16.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fb81c05c2a0 0x7fb81c05e760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:16.805 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb83006a350 0x7fb830109070 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:16.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 --2- 172.21.15.93:0/3940253018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83006b960 0x7fb830109560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:16.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 >> 172.21.15.93:0/3940253018 conn(0x7fb8300f1de0 msgr2=0x7fb8300fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:16.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 shutdown_connections 2021-11-20T09:01:16.806 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:16.805+0000 7fb835316700 1 -- 172.21.15.93:0/3940253018 wait complete. 2021-11-20T09:01:16.807 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 93 2021-11-20T09:01:16.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:16.795621+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:16.820 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:reconnect 2021-11-20T09:01:17.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: cluster 2021-11-20T09:01:15.587884+0000 mgr.smithi093.qdikkm (mgr.14162) 321 : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 204 B/s wr, 0 op/s 2021-11-20T09:01:17.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: cluster 2021-11-20T09:01:15.786977+0000 mon.smithi093 (mon.0) 881 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:01:17.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: cluster 2021-11-20T09:01:15.794122+0000 mon.smithi093 (mon.0) 882 : cluster 2021-11-20T09:01:17.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:boot 2021-11-20T09:01:17.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: cluster 2021 2021-11-20T09:01:17.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: -11-20T09:01:15.794168+0000 mon.smithi093 2021-11-20T09:01:17.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: (mon.0) 883 : cluster [INF] 2021-11-20T09:01:17.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: Standby daemon mds.cephfs.smithi102.hjodmi assigned to filesystem cephfs as rank 0 2021-11-20T09:01:17.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: cluster 2021-11-20T09:01:17.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: 2021-11-20T09:01: 2021-11-20T09:01:17.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: 15.794287+0000 mon.smithi093 (mon.0 2021-11-20T09:01:17.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: ) 884 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:01:17.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T09:01:17.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: -20T09:01:15.794514+0000 mon.smithi093 ( 2021-11-20T09:01:17.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: mon.0) 885 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:01:17.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: 2021-11-20T09:01:17.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: audit 2021-11-20T09:01:17.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: 2021-11-20T09:01 2021-11-20T09:01:17.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: :15.794617+0000 mon.smithi093 (mon. 2021-11-20T09:01:17.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: 0) 886 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T09:01:17.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T09:01:17.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: -20T09:01:15.797164+0000 mon.smithi093 2021-11-20T09:01:17.173 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: (mon.0) 887 : cluster [DBG] 2021-11-20T09:01:17.174 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:16 smithi102 conmon[36472]: fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:replay} 1 up:standby 2021-11-20T09:01:17.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: cluster 2021-11-20T09:01:15.587884+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T09:01:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: 14162) 321 : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 204 B/s wr, 0 op/s 2021-11-20T09:01:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: cluster 2021-11-20T09:01:15.786977+0000 mon.smithi093 (mon.0) 881 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:01:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: cluster 2021-11-20T09 2021-11-20T09:01:17.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: :01:15.794122+0000 mon.smithi093 (mon.0) 882 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:boot 2021-11-20T09:01:17.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: cluster 2021-11-20T09: 2021-11-20T09:01:17.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: 01:15.794168+0000 mon.smithi093 (mon.0) 883 : cluster [INF] Standby daemon mds.cephfs.smithi102.hjodmi assigned to filesystem cephfs as rank 0 2021-11-20T09:01:17.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: cluster 2021-11-20T09:01:15. 2021-11-20T09:01:17.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: 794287+0000 mon.smithi093 (mon.0) 884 : cluster [INF] 2021-11-20T09:01:17.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:01:17.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T09:01:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: T09:01:15.794514+0000 mon.smithi093 (mon 2021-11-20T09:01:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: .0) 885 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:01:17.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: audit 2021-11-20T09:01 2021-11-20T09:01:17.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: :15.794617+0000 mon.smithi093 (mon.0) 886 : audit [DBG] 2021-11-20T09:01:17.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.lhrlmh"}]: dispatch 2021-11-20T09:01:17.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: cluster 2021-11- 2021-11-20T09:01:17.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:16 smithi093 conmon[33522]: 20T09:01:15.797164+0000 mon.smithi093 (mon.0) 887 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:replay} 1 up:standby 2021-11-20T09:01:17.821 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:17.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- 172.21.15.93:0/2199818610 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f57c0 msgr2=0x7f30640f5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:17.948 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 --2- 172.21.15.93:0/2199818610 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f57c0 0x7f30640f5be0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f3060004660 tx=0x7f3060009b30).stop 2021-11-20T09:01:17.949 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- 172.21.15.93:0/2199818610 shutdown_connections 2021-11-20T09:01:17.949 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 --2- 172.21.15.93:0/2199818610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30640f6a60 0x7f30640f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:17.950 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 --2- 172.21.15.93:0/2199818610 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f57c0 0x7f30640f5be0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:17.951 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- 172.21.15.93:0/2199818610 >> 172.21.15.93:0/2199818610 conn(0x7f30640f0ec0 msgr2=0x7f30640f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:17.951 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- 172.21.15.93:0/2199818610 shutdown_connections 2021-11-20T09:01:17.951 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- 172.21.15.93:0/2199818610 wait complete. 2021-11-20T09:01:17.951 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 Processor -- start 2021-11-20T09:01:17.951 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- start start 2021-11-20T09:01:17.952 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30640f57c0 0x7f306405ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:17.952 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f6a60 0x7f306410bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:17.952 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306410c3f0 con 0x7f30640f6a60 2021-11-20T09:01:17.952 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.947+0000 7f306b57a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306410c530 con 0x7f30640f57c0 2021-11-20T09:01:17.953 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3068b15700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f6a60 0x7f306410bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:17.953 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3068b15700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f6a60 0x7f306410bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44696/0 (socket says 172.21.15.93:44696) 2021-11-20T09:01:17.953 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3068b15700 1 -- 172.21.15.93:0/575572010 learned_addr learned my addr 172.21.15.93:0/575572010 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:17.953 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3069316700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30640f57c0 0x7f306405ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:17.954 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3068b15700 1 -- 172.21.15.93:0/575572010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30640f57c0 msgr2=0x7f306405ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:17.954 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3068b15700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30640f57c0 0x7f306405ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:17.954 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3068b15700 1 -- 172.21.15.93:0/575572010 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3060005040 con 0x7f30640f6a60 2021-11-20T09:01:17.954 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3069316700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30640f57c0 0x7f306405ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:01:17.955 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3068b15700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f6a60 0x7f306410bf00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f305400b410 tx=0x7f3054006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:17.955 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3059ffb700 1 -- 172.21.15.93:0/575572010 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30540125f0 con 0x7f30640f6a60 2021-11-20T09:01:17.955 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3059ffb700 1 -- 172.21.15.93:0/575572010 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3054012c00 con 0x7f30640f6a60 2021-11-20T09:01:17.955 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f3059ffb700 1 -- 172.21.15.93:0/575572010 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3054002aa0 con 0x7f30640f6a60 2021-11-20T09:01:17.956 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306410c810 con 0x7f30640f6a60 2021-11-20T09:01:17.956 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.948+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306410cd80 con 0x7f30640f6a60 2021-11-20T09:01:17.956 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.949+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306405ea10 con 0x7f30640f6a60 2021-11-20T09:01:17.956 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.949+0000 7f3059ffb700 1 -- 172.21.15.93:0/575572010 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f305400f030 con 0x7f30640f6a60 2021-11-20T09:01:17.957 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.950+0000 7f3059ffb700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f305005c2a0 0x7f305005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:17.958 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.950+0000 7f3059ffb700 1 -- 172.21.15.93:0/575572010 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f3054076900 con 0x7f30640f6a60 2021-11-20T09:01:17.958 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.950+0000 7f3069316700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f305005c2a0 0x7f305005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:17.958 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.950+0000 7f3069316700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f305005c2a0 0x7f305005e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f3060009d60 tx=0x7f3060009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:17.958 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:17.953+0000 7f3059ffb700 1 -- 172.21.15.93:0/575572010 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3054049000 con 0x7f30640f6a60 2021-11-20T09:01:18.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.136+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3064027a00 con 0x7f30640f6a60 2021-11-20T09:01:18.135 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.136+0000 7f3059ffb700 1 -- 172.21.15.93:0/575572010 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+2186 (secure 0 0 0) 0x7f305404c620 con 0x7f30640f6a60 2021-11-20T09:01:18.136 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:18.137 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:17.797339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:18.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f305005c2a0 msgr2=0x7f305005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:18.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f305005c2a0 0x7f305005e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f3060009d60 tx=0x7f3060009b30).stop 2021-11-20T09:01:18.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f6a60 msgr2=0x7f306410bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:18.138 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f6a60 0x7f306410bf00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f305400b410 tx=0x7f3054006ef0).stop 2021-11-20T09:01:18.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 shutdown_connections 2021-11-20T09:01:18.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f305005c2a0 0x7f305005e760 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:18.139 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30640f57c0 0x7f306405ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:18.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 --2- 172.21.15.93:0/575572010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30640f6a60 0x7f306410bf00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:18.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 >> 172.21.15.93:0/575572010 conn(0x7f30640f0ec0 msgr2=0x7f30640fa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:18.140 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 shutdown_connections 2021-11-20T09:01:18.141 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:18.138+0000 7f306b57a700 1 -- 172.21.15.93:0/575572010 wait complete. 2021-11-20T09:01:18.141 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 94 2021-11-20T09:01:18.148 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:17 smithi093 conmon[33522]: cluster 2021-11-20T09:01:16.798179+0000 mon.smithi093 (mon.0) 888 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:reconnect 2021-11-20T09:01:18.149 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:17 smithi093 conmon[33522]: 2021-11-20T09:01:18.149 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:17 smithi093 conmon[33522]: cluster 2021-11-20T09:01:16.798238+0000 mon.smithi093 (mon.0) 889 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:reconnect} 1 up:standby 2021-11-20T09:01:18.149 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:17 smithi093 conmon[33522]: audit 2021-11-20T09:01:16.803060+0000 mon.smithi093 ( 2021-11-20T09:01:18.150 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:17 smithi093 conmon[33522]: mon.0) 890 : audit [DBG] from='client.? 172.21.15.93:0/3940253018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:18.150 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:17 smithi093 conmon[33522]: cluster 2021-11-20T09:01:16.818576+0000 mds.cephfs.smithi102.hjodmi (mds. 2021-11-20T09:01:18.150 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:17 smithi093 conmon[33522]: 0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0189996 2021-11-20T09:01:18.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:17.797339+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:18.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:rejoin 2021-11-20T09:01:18.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: cluster 2021-11-20T09:01:16.798179+0000 2021-11-20T09:01:18.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: mon.smithi093 (mon.0) 888 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:reconnect 2021-11-20T09:01:18.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: cluster 2021-11-20T09:01:16.798238+0000 mon.smithi093 (mon.0) 889 : cluster 2021-11-20T09:01:18.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:reconnect} 1 up:standby 2021-11-20T09:01:18.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: audit 2021-11-20T09:01:18.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: 2021-11-20T09:01:16.803060+0000 2021-11-20T09:01:18.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: mon.smithi093 (mon.0) 890 : audit [DBG] 2021-11-20T09:01:18.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: from='client.? 172.21.15.93:0/3940253018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:18.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: cluster 2021 2021-11-20T09:01:18.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: -11-20T09:01:16 2021-11-20T09:01:18.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: .818576+0000 mds.cephfs.smithi102.hjodmi (mds.0) 2021-11-20T09:01:18.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:17 smithi102 conmon[36472]: 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0189996 2021-11-20T09:01:19.154 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:19.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:18 smithi102 conmon[36472]: cluster 2021-11-20T09:01:17.588508+0000 mgr.smithi093.qdikkm (mgr.14162) 322 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T09:01:19.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:18 smithi102 conmon[36472]: cluster 2021-11-20T09:01:17.804100+0000 mon.smithi093 (mon.0) 891 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:rejoin 2021-11-20T09:01:19.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:18 smithi102 conmon[36472]: cluster 2021-11-20T09:01:17.804151+0000 mon.smithi093 (mon.0 2021-11-20T09:01:19.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:18 smithi102 conmon[36472]: ) 892 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:rejoin} 1 up:standby 2021-11-20T09:01:19.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:18 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T09:01:19.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:18 smithi102 conmon[36472]: 09:01:18.137533+0000 mon.smithi093 (mon. 2021-11-20T09:01:19.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:18 smithi102 conmon[36472]: 0) 893 : audit [DBG] from='client.? 172.21.15.93:0/575572010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:19.182 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:18 smithi093 conmon[33522]: cluster 2021-11-20T09:01:17.588508+0000 mgr.smithi093.qdikkm (mgr.14162) 322 : cluster 2021-11-20T09:01:19.183 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:18 smithi093 conmon[33522]: [DBG] pgmap v301: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T09:01:19.183 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:18 smithi093 conmon[33522]: cluster 2021-11-20T09:01:17.804100+0000 mon.smithi093 (mon.0) 891 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:rejoin 2021-11-20T09:01:19.183 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:18 smithi093 conmon[33522]: cluster 2021-11-20T09:01:17.804151+0000 2021-11-20T09:01:19.184 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:18 smithi093 conmon[33522]: mon.smithi093 (mon.0) 892 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:rejoin} 1 up:standby 2021-11-20T09:01:19.184 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:18 smithi093 conmon[33522]: audit 2021-11-20T09:01: 2021-11-20T09:01:19.184 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:18 smithi093 conmon[33522]: 18.137533+0000 mon.smithi093 (mon.0) 893 : audit [DBG] from='client.? 172.21.15.93:0/575572010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:19.300 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 -- 172.21.15.93:0/1832153806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 msgr2=0x7f55bc0f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:19.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 --2- 172.21.15.93:0/1832153806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc0f6c70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f55a4004660 tx=0x7f55a4009b30).stop 2021-11-20T09:01:19.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 -- 172.21.15.93:0/1832153806 shutdown_connections 2021-11-20T09:01:19.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 --2- 172.21.15.93:0/1832153806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc0f6c70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:19.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 --2- 172.21.15.93:0/1832153806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55bc0f5570 0x7f55bc0f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:19.301 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 -- 172.21.15.93:0/1832153806 >> 172.21.15.93:0/1832153806 conn(0x7f55bc0f0cb0 msgr2=0x7f55bc0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:19.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 -- 172.21.15.93:0/1832153806 shutdown_connections 2021-11-20T09:01:19.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.299+0000 7f55c13c0700 1 -- 172.21.15.93:0/1832153806 wait complete. 2021-11-20T09:01:19.302 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55c13c0700 1 Processor -- start 2021-11-20T09:01:19.303 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55c13c0700 1 -- start start 2021-11-20T09:01:19.303 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55c13c0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55bc0f5570 0x7f55bc060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:19.303 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55c13c0700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc10bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:19.303 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55c13c0700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc10c160 con 0x7f55bc0f6810 2021-11-20T09:01:19.304 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55c13c0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc10c2a0 con 0x7f55bc0f5570 2021-11-20T09:01:19.304 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55ba7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc10bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:19.304 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55ba7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc10bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44708/0 (socket says 172.21.15.93:44708) 2021-11-20T09:01:19.304 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.300+0000 7f55ba7fc700 1 -- 172.21.15.93:0/820930938 learned_addr learned my addr 172.21.15.93:0/820930938 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:19.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55baffd700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55bc0f5570 0x7f55bc060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:19.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55ba7fc700 1 -- 172.21.15.93:0/820930938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55bc0f5570 msgr2=0x7f55bc060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:19.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55ba7fc700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55bc0f5570 0x7f55bc060550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:19.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55ba7fc700 1 -- 172.21.15.93:0/820930938 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a4005040 con 0x7f55bc0f6810 2021-11-20T09:01:19.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55baffd700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55bc0f5570 0x7f55bc060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:01:19.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55ba7fc700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc10bc70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f55a4009830 tx=0x7f55a4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:19.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55b37fe700 1 -- 172.21.15.93:0/820930938 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55a4018070 con 0x7f55bc0f6810 2021-11-20T09:01:19.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc10c4a0 con 0x7f55bc0f6810 2021-11-20T09:01:19.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55b37fe700 1 -- 172.21.15.93:0/820930938 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55a4013410 con 0x7f55bc0f6810 2021-11-20T09:01:19.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55b37fe700 1 -- 172.21.15.93:0/820930938 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55a401c3c0 con 0x7f55bc0f6810 2021-11-20T09:01:19.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.301+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc10ca00 con 0x7f55bc0f6810 2021-11-20T09:01:19.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.302+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc027a00 con 0x7f55bc0f6810 2021-11-20T09:01:19.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.302+0000 7f55b37fe700 1 -- 172.21.15.93:0/820930938 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f55a401d420 con 0x7f55bc0f6810 2021-11-20T09:01:19.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.303+0000 7f55b37fe700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f55a805c2a0 0x7f55a805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:19.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.303+0000 7f55b37fe700 1 -- 172.21.15.93:0/820930938 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f55a40781b0 con 0x7f55bc0f6810 2021-11-20T09:01:19.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.303+0000 7f55baffd700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f55a805c2a0 0x7f55a805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:19.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.303+0000 7f55baffd700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f55a805c2a0 0x7f55a805e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f55bc0f6670 tx=0x7f55ac004d00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:19.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.307+0000 7f55b37fe700 1 -- 172.21.15.93:0/820930938 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55a404a910 con 0x7f55bc0f6810 2021-11-20T09:01:19.490 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.491+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55bc0fdd00 con 0x7f55bc0f6810 2021-11-20T09:01:19.491 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.492+0000 7f55b37fe700 1 -- 172.21.15.93:0/820930938 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f55a404df30 con 0x7f55bc0f6810 2021-11-20T09:01:19.492 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:19.492 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:19.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f55a805c2a0 msgr2=0x7f55a805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:19.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f55a805c2a0 0x7f55a805e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f55bc0f6670 tx=0x7f55ac004d00).stop 2021-11-20T09:01:19.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 msgr2=0x7f55bc10bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:19.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc10bc70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f55a4009830 tx=0x7f55a4009b30).stop 2021-11-20T09:01:19.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 shutdown_connections 2021-11-20T09:01:19.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f55a805c2a0 0x7f55a805e760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:19.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55bc0f5570 0x7f55bc060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:19.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 --2- 172.21.15.93:0/820930938 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55bc0f6810 0x7f55bc10bc70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:19.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.493+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 >> 172.21.15.93:0/820930938 conn(0x7f55bc0f0cb0 msgr2=0x7f55bc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:19.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.494+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 shutdown_connections 2021-11-20T09:01:19.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:19.494+0000 7f55c13c0700 1 -- 172.21.15.93:0/820930938 wait complete. 2021-11-20T09:01:19.506 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 95 2021-11-20T09:01:19.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:19.510 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:20.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:19 smithi102 conmon[36472]: cluster 2021-11-20T09:01:18.805325+0000 mon.smithi093 (mon.0) 894 : 2021-11-20T09:01:20.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:19 smithi102 conmon[36472]: cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:clientreplay 2021-11-20T09:01:20.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:19 smithi102 conmon[36472]: cluster 2021-11-20T09:01:18.805401+0000 mon.smithi093 (mon.0) 895 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:01:20.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:19 smithi102 conmon[36472]: audit 2021-11-20T09:01:20.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:19 smithi102 conmon[36472]: 2021-11-20T09:01:19.492905+0000 mon.smithi093 (mon.0) 896 : audit [DBG] from='client.? 172.21.15.93:0/820930938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:20.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:19 smithi093 conmon[33522]: cluster 2021-11-20T09:01:18. 2021-11-20T09:01:20.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:19 smithi093 conmon[33522]: 805325+0000 mon.smithi093 (mon.0) 894 : cluster [DBG] mds.? [v2:172.21.15.102:6826/2769927812,v1:172.21.15.102:6827/2769927812] up:clientreplay 2021-11-20T09:01:20.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:19 smithi093 conmon[33522]: cluster 2021-11-20T09:01:18.805401+0000 mon.smithi093 (mon.0 2021-11-20T09:01:20.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:19 smithi093 conmon[33522]: ) 895 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:01:20.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:19 smithi093 conmon[33522]: audit 2021-11-20T09:01:20.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:19 smithi093 conmon[33522]: 2021-11-20T09:01:19.492905+0000 mon.smithi093 (mon.0 2021-11-20T09:01:20.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:19 smithi093 conmon[33522]: ) 896 : audit [DBG] from='client.? 172.21.15.93:0/820930938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:20.512 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:20.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- 172.21.15.93:0/3176215146 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f4780 msgr2=0x7f42280f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:20.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 --2- 172.21.15.93:0/3176215146 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f4780 0x7f42280f4c00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f4210004660 tx=0x7f4210009b30).stop 2021-11-20T09:01:20.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- 172.21.15.93:0/3176215146 shutdown_connections 2021-11-20T09:01:20.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 --2- 172.21.15.93:0/3176215146 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f4780 0x7f42280f4c00 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:20.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 --2- 172.21.15.93:0/3176215146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42280f7d90 0x7f42280f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:20.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- 172.21.15.93:0/3176215146 >> 172.21.15.93:0/3176215146 conn(0x7f42280f04a0 msgr2=0x7f42280f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:20.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- 172.21.15.93:0/3176215146 shutdown_connections 2021-11-20T09:01:20.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- 172.21.15.93:0/3176215146 wait complete. 2021-11-20T09:01:20.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 Processor -- start 2021-11-20T09:01:20.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- start start 2021-11-20T09:01:20.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42280f4780 0x7f42280f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:20.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f7d90 0x7f42280f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:20.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280f7900 con 0x7f42280f7d90 2021-11-20T09:01:20.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f422c8a7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280f5ea0 con 0x7f42280f4780 2021-11-20T09:01:20.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.640+0000 7f421ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f7d90 0x7f42280f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:20.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421ffff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f7d90 0x7f42280f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44722/0 (socket says 172.21.15.93:44722) 2021-11-20T09:01:20.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421ffff700 1 -- 172.21.15.93:0/521546168 learned_addr learned my addr 172.21.15.93:0/521546168 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:20.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f422659c700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42280f4780 0x7f42280f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:20.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421ffff700 1 -- 172.21.15.93:0/521546168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42280f4780 msgr2=0x7f42280f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:20.648 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421ffff700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42280f4780 0x7f42280f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:20.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421ffff700 1 -- 172.21.15.93:0/521546168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4210005040 con 0x7f42280f7d90 2021-11-20T09:01:20.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421ffff700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f7d90 0x7f42280f59b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4210004e30 tx=0x7f4210009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:20.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421effd700 1 -- 172.21.15.93:0/521546168 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4210018070 con 0x7f42280f7d90 2021-11-20T09:01:20.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42280f60c0 con 0x7f42280f7d90 2021-11-20T09:01:20.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421effd700 1 -- 172.21.15.93:0/521546168 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4210013410 con 0x7f42280f7d90 2021-11-20T09:01:20.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f421effd700 1 -- 172.21.15.93:0/521546168 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f421001de20 con 0x7f42280f7d90 2021-11-20T09:01:20.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.641+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42280f6620 con 0x7f42280f7d90 2021-11-20T09:01:20.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.642+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4228046330 con 0x7f42280f7d90 2021-11-20T09:01:20.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.642+0000 7f421effd700 1 -- 172.21.15.93:0/521546168 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f421001d420 con 0x7f42280f7d90 2021-11-20T09:01:20.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.643+0000 7f421effd700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f421405c2f0 0x7f421405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:20.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.643+0000 7f421effd700 1 -- 172.21.15.93:0/521546168 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f4210078110 con 0x7f42280f7d90 2021-11-20T09:01:20.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.643+0000 7f422659c700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f421405c2f0 0x7f421405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:20.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.643+0000 7f422659c700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f421405c2f0 0x7f421405e7b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f422805f7c0 tx=0x7f421800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:20.682 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.647+0000 7f421effd700 1 -- 172.21.15.93:0/521546168 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f421004a7b0 con 0x7f42280f7d90 2021-11-20T09:01:20.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.830+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4228027a00 con 0x7f42280f7d90 2021-11-20T09:01:20.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.831+0000 7f421effd700 1 -- 172.21.15.93:0/521546168 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f4228027a00 con 0x7f42280f7d90 2021-11-20T09:01:20.830 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:20.830 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:20.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f421405c2f0 msgr2=0x7f421405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:20.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f421405c2f0 0x7f421405e7b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f422805f7c0 tx=0x7f421800b040).stop 2021-11-20T09:01:20.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f7d90 msgr2=0x7f42280f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:20.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f7d90 0x7f42280f59b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4210004e30 tx=0x7f4210009b30).stop 2021-11-20T09:01:20.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 shutdown_connections 2021-11-20T09:01:20.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f421405c2f0 0x7f421405e7b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:20.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42280f4780 0x7f42280f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:20.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 --2- 172.21.15.93:0/521546168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42280f7d90 0x7f42280f59b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:20.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 >> 172.21.15.93:0/521546168 conn(0x7f42280f04a0 msgr2=0x7f42280fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:20.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 shutdown_connections 2021-11-20T09:01:20.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:20.832+0000 7f422c8a7700 1 -- 172.21.15.93:0/521546168 wait complete. 2021-11-20T09:01:20.834 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 96 2021-11-20T09:01:20.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:20.844 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:21.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:20 smithi102 conmon[36472]: cluster 2021-11-20T09:01:19.588979+0000 mgr.smithi093.qdikkm (mgr.14162) 2021-11-20T09:01:21.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:20 smithi102 conmon[36472]: 323 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T09:01:21.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:20 smithi102 conmon[36472]: cluster 2021-11-20T09:01:19.812278+0000 mon.smithi093 (mon.0) 897 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:standby 2021-11-20T09:01:21.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:20 smithi102 conmon[36472]: cluster 2021 2021-11-20T09:01:21.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:20 smithi102 conmon[36472]: -11-20T09:01:19.812324+0000 mon.smithi093 (mon.0) 898 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:01:21.210 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:20 smithi093 conmon[33522]: cluster 2021-11-20T09:01:19.588979+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T09:01:21.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:20 smithi093 conmon[33522]: 14162) 323 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 3.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T09:01:21.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:20 smithi093 conmon[33522]: cluster 2021-11-20T09:01:19.812278+0000 mon.smithi093 (mon.0) 897 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:standby 2021-11-20T09:01:21.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:20 smithi093 conmon[33522]: cluster 2021 2021-11-20T09:01:21.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:20 smithi093 conmon[33522]: -11-20T09:01:19.812324+0000 mon.smithi093 (mon.0) 898 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.hjodmi=up:clientreplay} 1 up:standby 2021-11-20T09:01:21.846 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:21.969 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 -- 172.21.15.93:0/178832143 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f5580 msgr2=0x7fc19c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:21.969 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 --2- 172.21.15.93:0/178832143 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f5580 0x7fc19c0f59a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7fc198004660 tx=0x7fc198009b30).stop 2021-11-20T09:01:21.974 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 -- 172.21.15.93:0/178832143 shutdown_connections 2021-11-20T09:01:21.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 --2- 172.21.15.93:0/178832143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19c0f6820 0x7fc19c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:21.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 --2- 172.21.15.93:0/178832143 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f5580 0x7fc19c0f59a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:21.977 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 -- 172.21.15.93:0/178832143 >> 172.21.15.93:0/178832143 conn(0x7fc19c0f0ce0 msgr2=0x7fc19c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:21.977 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 -- 172.21.15.93:0/178832143 shutdown_connections 2021-11-20T09:01:21.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 -- 172.21.15.93:0/178832143 wait complete. 2021-11-20T09:01:21.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.967+0000 7fc1a4529700 1 Processor -- start 2021-11-20T09:01:21.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a4529700 1 -- start start 2021-11-20T09:01:21.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a4529700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19c0f5580 0x7fc19c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:21.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a4529700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f6820 0x7fc19c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:21.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a4529700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c107950 con 0x7fc19c0f6820 2021-11-20T09:01:21.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a4529700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c107a90 con 0x7fc19c0f5580 2021-11-20T09:01:21.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a1ac4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f6820 0x7fc19c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:21.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a1ac4700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f6820 0x7fc19c1073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44734/0 (socket says 172.21.15.93:44734) 2021-11-20T09:01:21.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a1ac4700 1 -- 172.21.15.93:0/2641180181 learned_addr learned my addr 172.21.15.93:0/2641180181 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:21.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a22c5700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19c0f5580 0x7fc19c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:21.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a1ac4700 1 -- 172.21.15.93:0/2641180181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19c0f5580 msgr2=0x7fc19c106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:21.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a1ac4700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19c0f5580 0x7fc19c106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:21.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a1ac4700 1 -- 172.21.15.93:0/2641180181 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198005040 con 0x7fc19c0f6820 2021-11-20T09:01:21.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a22c5700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19c0f5580 0x7fc19c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T09:01:21.990 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a1ac4700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f6820 0x7fc19c1073a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fc18c00c530 tx=0x7fc18c007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:21.990 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc192ffd700 1 -- 172.21.15.93:0/2641180181 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc18c00f650 con 0x7fc19c0f6820 2021-11-20T09:01:21.991 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc192ffd700 1 -- 172.21.15.93:0/2641180181 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc18c00fc60 con 0x7fc19c0f6820 2021-11-20T09:01:21.991 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c10c530 con 0x7fc19c0f6820 2021-11-20T09:01:21.991 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc192ffd700 1 -- 172.21.15.93:0/2641180181 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc18c012d30 con 0x7fc19c0f6820 2021-11-20T09:01:21.992 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.968+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c10caf0 con 0x7fc19c0f6820 2021-11-20T09:01:21.992 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.969+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c0fdd10 con 0x7fc19c0f6820 2021-11-20T09:01:21.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.970+0000 7fc192ffd700 1 -- 172.21.15.93:0/2641180181 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc18c010070 con 0x7fc19c0f6820 2021-11-20T09:01:21.993 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.970+0000 7fc192ffd700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc18805c2a0 0x7fc18805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:21.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.970+0000 7fc192ffd700 1 -- 172.21.15.93:0/2641180181 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fc18c076170 con 0x7fc19c0f6820 2021-11-20T09:01:21.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.970+0000 7fc1a22c5700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc18805c2a0 0x7fc18805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:21.994 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.970+0000 7fc1a22c5700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc18805c2a0 0x7fc18805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fc198004e30 tx=0x7fc198009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:21.996 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:21.974+0000 7fc192ffd700 1 -- 172.21.15.93:0/2641180181 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc18c0452b0 con 0x7fc19c0f6820 2021-11-20T09:01:22.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.156+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc19c10d1f0 con 0x7fc19c0f6820 2021-11-20T09:01:22.156 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.157+0000 7fc192ffd700 1 -- 172.21.15.93:0/2641180181 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2191 (secure 0 0 0) 0x7fc18c0488d0 con 0x7fc19c0f6820 2021-11-20T09:01:22.157 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:22.158 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:22.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc18805c2a0 msgr2=0x7fc18805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:22.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc18805c2a0 0x7fc18805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fc198004e30 tx=0x7fc198009b30).stop 2021-11-20T09:01:22.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f6820 msgr2=0x7fc19c1073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:22.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f6820 0x7fc19c1073a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fc18c00c530 tx=0x7fc18c007e20).stop 2021-11-20T09:01:22.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 shutdown_connections 2021-11-20T09:01:22.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fc18805c2a0 0x7fc18805e760 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:22.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19c0f5580 0x7fc19c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:22.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 --2- 172.21.15.93:0/2641180181 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc19c0f6820 0x7fc19c1073a0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:22.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.158+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 >> 172.21.15.93:0/2641180181 conn(0x7fc19c0f0ce0 msgr2=0x7fc19c05fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:22.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.159+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 shutdown_connections 2021-11-20T09:01:22.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:22.159+0000 7fc1a4529700 1 -- 172.21.15.93:0/2641180181 wait complete. 2021-11-20T09:01:22.163 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 96 2021-11-20T09:01:22.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:21 smithi102 conmon[36472]: audit 2021-11-20T09:01:20.831573+0000 mon.smithi093 (mon.0) 899 : audit [DBG] from='client.? 172.21.15.93:0/521546168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:22.168 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:21 smithi093 conmon[33522]: audit 2021-11-20T09:01:20.831573+0000 mon.smithi093 (mon.0) 899 : audit [DBG] from='client.? 172.21.15.93:0/521546168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:22.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:22.170 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:23.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:22 smithi102 conmon[36472]: cluster 2021-11-20T09:01:21.589459+0000 mgr.smithi093.qdikkm (mgr.14162) 324 : cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:23.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:22 smithi102 conmon[36472]: audit 2021-11-20T09:01: 2021-11-20T09:01:23.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:22 smithi102 conmon[36472]: 22.157817+0000 mon.smithi093 (mon.0) 900 : audit [DBG] from='client.? 172.21.15.93:0/2641180181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:23.171 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:23.199 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:22 smithi093 conmon[33522]: cluster 2021-11-20T09:01:21.589459+0000 mgr.smithi093.qdikkm (mgr.14162 2021-11-20T09:01:23.199 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:22 smithi093 conmon[33522]: ) 324 : cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:23.199 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:22 smithi093 conmon[33522]: audit 2021-11-20T09:01:22.157817+0000 mon.smithi093 (mon.0) 900 : audit [DBG] from='client.? 172.21.15.93:0/2641180181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:23.304 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 -- 172.21.15.93:0/3918292760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f5580 msgr2=0x7f2a480f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:23.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/3918292760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f5580 0x7f2a480f59a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f2a38004660 tx=0x7f2a38009b30).stop 2021-11-20T09:01:23.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 -- 172.21.15.93:0/3918292760 shutdown_connections 2021-11-20T09:01:23.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/3918292760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a480f6820 0x7f2a480f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:23.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/3918292760 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f5580 0x7f2a480f59a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:23.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 -- 172.21.15.93:0/3918292760 >> 172.21.15.93:0/3918292760 conn(0x7f2a480f0ce0 msgr2=0x7f2a480f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:23.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 -- 172.21.15.93:0/3918292760 shutdown_connections 2021-11-20T09:01:23.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.303+0000 7f2a4e63b700 1 -- 172.21.15.93:0/3918292760 wait complete. 2021-11-20T09:01:23.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a4e63b700 1 Processor -- start 2021-11-20T09:01:23.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a4e63b700 1 -- start start 2021-11-20T09:01:23.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a4e63b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a480f5580 0x7f2a48060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:23.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a4e63b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f6820 0x7f2a4810bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:23.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a4e63b700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4810c190 con 0x7f2a480f6820 2021-11-20T09:01:23.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a4e63b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4810c2d0 con 0x7f2a480f5580 2021-11-20T09:01:23.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a477fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f6820 0x7f2a4810bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:23.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a477fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f6820 0x7f2a4810bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44746/0 (socket says 172.21.15.93:44746) 2021-11-20T09:01:23.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a477fe700 1 -- 172.21.15.93:0/707431423 learned_addr learned my addr 172.21.15.93:0/707431423 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:23.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a47fff700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a480f5580 0x7f2a48060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:23.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a477fe700 1 -- 172.21.15.93:0/707431423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a480f5580 msgr2=0x7f2a48060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:23.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.304+0000 7f2a477fe700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a480f5580 0x7f2a48060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:23.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a477fe700 1 -- 172.21.15.93:0/707431423 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a38005040 con 0x7f2a480f6820 2021-11-20T09:01:23.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a47fff700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a480f5580 0x7f2a48060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:01:23.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a477fe700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f6820 0x7f2a4810bca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f2a3c00c560 tx=0x7f2a3c007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:23.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a44ff9700 1 -- 172.21.15.93:0/707431423 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a3c00f650 con 0x7f2a480f6820 2021-11-20T09:01:23.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a44ff9700 1 -- 172.21.15.93:0/707431423 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a3c00fc60 con 0x7f2a480f6820 2021-11-20T09:01:23.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4810c530 con 0x7f2a480f6820 2021-11-20T09:01:23.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a44ff9700 1 -- 172.21.15.93:0/707431423 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a3c012c80 con 0x7f2a480f6820 2021-11-20T09:01:23.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.305+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4810caf0 con 0x7f2a480f6820 2021-11-20T09:01:23.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.306+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a48027a00 con 0x7f2a480f6820 2021-11-20T09:01:23.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.306+0000 7f2a44ff9700 1 -- 172.21.15.93:0/707431423 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2a3c010070 con 0x7f2a480f6820 2021-11-20T09:01:23.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.307+0000 7f2a44ff9700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2a3005c2a0 0x7f2a3005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:23.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.307+0000 7f2a44ff9700 1 -- 172.21.15.93:0/707431423 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f2a3c076160 con 0x7f2a480f6820 2021-11-20T09:01:23.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.307+0000 7f2a47fff700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2a3005c2a0 0x7f2a3005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:23.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.307+0000 7f2a47fff700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2a3005c2a0 0x7f2a3005e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2a38009d60 tx=0x7f2a38009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:23.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.310+0000 7f2a44ff9700 1 -- 172.21.15.93:0/707431423 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a3c0452b0 con 0x7f2a480f6820 2021-11-20T09:01:23.494 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.495+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a480fdd10 con 0x7f2a480f6820 2021-11-20T09:01:23.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.495+0000 7f2a44ff9700 1 -- 172.21.15.93:0/707431423 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f2a3c04c2f0 con 0x7f2a480f6820 2021-11-20T09:01:23.495 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:23.496 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:23.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2a3005c2a0 msgr2=0x7f2a3005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:23.497 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2a3005c2a0 0x7f2a3005e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2a38009d60 tx=0x7f2a38009b30).stop 2021-11-20T09:01:23.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f6820 msgr2=0x7f2a4810bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:23.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f6820 0x7f2a4810bca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f2a3c00c560 tx=0x7f2a3c007e20).stop 2021-11-20T09:01:23.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 shutdown_connections 2021-11-20T09:01:23.498 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f2a3005c2a0 0x7f2a3005e760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:23.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a480f5580 0x7f2a48060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:23.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 --2- 172.21.15.93:0/707431423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a480f6820 0x7f2a4810bca0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:23.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 >> 172.21.15.93:0/707431423 conn(0x7f2a480f0ce0 msgr2=0x7f2a480fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:23.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 shutdown_connections 2021-11-20T09:01:23.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:23.497+0000 7f2a4e63b700 1 -- 172.21.15.93:0/707431423 wait complete. 2021-11-20T09:01:23.500 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 96 2021-11-20T09:01:23.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:23.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:24.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:23 smithi102 conmon[36472]: audit 2021-11-20T09:01:23.496462+0000 mon.smithi093 (mon.0) 901 : audit [DBG] from='client.? 172.21.15.93:0/707431423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:24.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:23 smithi093 conmon[33522]: audit 2021-11-20T09:01:23.496462+0000 mon.smithi093 (mon. 2021-11-20T09:01:24.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:23 smithi093 conmon[33522]: 0) 901 : audit [DBG] from='client.? 172.21.15.93:0/707431423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:24.510 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:24.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 -- 172.21.15.93:0/2113154624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f54d0 msgr2=0x7fe93c0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:24.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 --2- 172.21.15.93:0/2113154624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f54d0 0x7fe93c0f58f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fe930004660 tx=0x7fe930009b30).stop 2021-11-20T09:01:24.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 -- 172.21.15.93:0/2113154624 shutdown_connections 2021-11-20T09:01:24.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 --2- 172.21.15.93:0/2113154624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe93c0f6770 0x7fe93c0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:24.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 --2- 172.21.15.93:0/2113154624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f54d0 0x7fe93c0f58f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:24.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 -- 172.21.15.93:0/2113154624 >> 172.21.15.93:0/2113154624 conn(0x7fe93c0f0c30 msgr2=0x7fe93c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:24.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 -- 172.21.15.93:0/2113154624 shutdown_connections 2021-11-20T09:01:24.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.635+0000 7fe940f31700 1 -- 172.21.15.93:0/2113154624 wait complete. 2021-11-20T09:01:24.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe940f31700 1 Processor -- start 2021-11-20T09:01:24.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe940f31700 1 -- start start 2021-11-20T09:01:24.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe940f31700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe93c0f54d0 0x7fe93c109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:24.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe940f31700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f6770 0x7fe93c10bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:24.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe940f31700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c060540 con 0x7fe93c0f6770 2021-11-20T09:01:24.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe940f31700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c060680 con 0x7fe93c0f54d0 2021-11-20T09:01:24.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe93affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f6770 0x7fe93c10bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:24.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe93affd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f6770 0x7fe93c10bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44756/0 (socket says 172.21.15.93:44756) 2021-11-20T09:01:24.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe93affd700 1 -- 172.21.15.93:0/2536960872 learned_addr learned my addr 172.21.15.93:0/2536960872 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:24.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.636+0000 7fe93b7fe700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe93c0f54d0 0x7fe93c109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:24.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe93affd700 1 -- 172.21.15.93:0/2536960872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe93c0f54d0 msgr2=0x7fe93c109a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:24.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe93affd700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe93c0f54d0 0x7fe93c109a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:24.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe93affd700 1 -- 172.21.15.93:0/2536960872 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930005040 con 0x7fe93c0f6770 2021-11-20T09:01:24.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe93affd700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f6770 0x7fe93c10bf80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fe92c0026d0 tx=0x7fe92c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:24.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe923fff700 1 -- 172.21.15.93:0/2536960872 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92c0079c0 con 0x7fe93c0f6770 2021-11-20T09:01:24.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe923fff700 1 -- 172.21.15.93:0/2536960872 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe92c00f040 con 0x7fe93c0f6770 2021-11-20T09:01:24.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c10c530 con 0x7fe93c0f6770 2021-11-20T09:01:24.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe923fff700 1 -- 172.21.15.93:0/2536960872 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92c0148b0 con 0x7fe93c0f6770 2021-11-20T09:01:24.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.637+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c10caf0 con 0x7fe93c0f6770 2021-11-20T09:01:24.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.638+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c045ad0 con 0x7fe93c0f6770 2021-11-20T09:01:24.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.638+0000 7fe923fff700 1 -- 172.21.15.93:0/2536960872 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe92c014a10 con 0x7fe93c0f6770 2021-11-20T09:01:24.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.638+0000 7fe923fff700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe92405c300 0x7fe92405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:24.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.639+0000 7fe923fff700 1 -- 172.21.15.93:0/2536960872 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fe92c076390 con 0x7fe93c0f6770 2021-11-20T09:01:24.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.639+0000 7fe93b7fe700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe92405c300 0x7fe92405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:24.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.639+0000 7fe93b7fe700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe92405c300 0x7fe92405e7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fe93c0601c0 tx=0x7fe930009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:24.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.642+0000 7fe923fff700 1 -- 172.21.15.93:0/2536960872 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe92c048af0 con 0x7fe93c0f6770 2021-11-20T09:01:24.830 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.830+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe93c027a00 con 0x7fe93c0f6770 2021-11-20T09:01:24.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.830+0000 7fe923fff700 1 -- 172.21.15.93:0/2536960872 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2191 (secure 0 0 0) 0x7fe92c04c110 con 0x7fe93c0f6770 2021-11-20T09:01:24.832 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:24.832 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:24.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe92405c300 msgr2=0x7fe92405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:24.833 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe92405c300 0x7fe92405e7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fe93c0601c0 tx=0x7fe930009b30).stop 2021-11-20T09:01:24.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f6770 msgr2=0x7fe93c10bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:24.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f6770 0x7fe93c10bf80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fe92c0026d0 tx=0x7fe92c009de0).stop 2021-11-20T09:01:24.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 shutdown_connections 2021-11-20T09:01:24.834 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe92405c300 0x7fe92405e7c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:24.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe93c0f54d0 0x7fe93c109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:24.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 --2- 172.21.15.93:0/2536960872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe93c0f6770 0x7fe93c10bf80 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:24.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 >> 172.21.15.93:0/2536960872 conn(0x7fe93c0f0c30 msgr2=0x7fe93c0fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:24.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 shutdown_connections 2021-11-20T09:01:24.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:24.832+0000 7fe940f31700 1 -- 172.21.15.93:0/2536960872 wait complete. 2021-11-20T09:01:24.838 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 96 2021-11-20T09:01:24.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:24.844 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:25.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:24 smithi102 conmon[36472]: cluster 2021-11-20T09:01:23.589885+0000 mgr.smithi093.qdikkm (mgr.14162) 325 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:25.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:24 smithi093 conmon[33522]: cluster 2021-11-20T09:01:23.589885+0000 mgr.smithi093.qdikkm (mgr.14162) 325 : cluster 2021-11-20T09:01:25.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:24 smithi093 conmon[33522]: [DBG] pgmap v304: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:25.846 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:25.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 -- 172.21.15.93:0/2677920755 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f55a0 msgr2=0x7f76cc0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:25.975 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 --2- 172.21.15.93:0/2677920755 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f55a0 0x7f76cc0f59c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f76bc0099e0 tx=0x7f76bc004d10).stop 2021-11-20T09:01:25.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 -- 172.21.15.93:0/2677920755 shutdown_connections 2021-11-20T09:01:25.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 --2- 172.21.15.93:0/2677920755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76cc0f6840 0x7f76cc0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:25.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 --2- 172.21.15.93:0/2677920755 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f55a0 0x7f76cc0f59c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:25.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 -- 172.21.15.93:0/2677920755 >> 172.21.15.93:0/2677920755 conn(0x7f76cc0f0ce0 msgr2=0x7f76cc0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:25.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 -- 172.21.15.93:0/2677920755 shutdown_connections 2021-11-20T09:01:25.977 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.972+0000 7f76d1289700 1 -- 172.21.15.93:0/2677920755 wait complete. 2021-11-20T09:01:25.977 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76d1289700 1 Processor -- start 2021-11-20T09:01:25.977 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76d1289700 1 -- start start 2021-11-20T09:01:25.977 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76d1289700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76cc0f55a0 0x7f76cc105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:25.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76d1289700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f6840 0x7f76cc1063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:25.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76d1289700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc106950 con 0x7f76cc0f6840 2021-11-20T09:01:25.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76d1289700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc106a90 con 0x7f76cc0f55a0 2021-11-20T09:01:25.978 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76ca7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f6840 0x7f76cc1063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:25.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76ca7fc700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f6840 0x7f76cc1063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44766/0 (socket says 172.21.15.93:44766) 2021-11-20T09:01:25.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76ca7fc700 1 -- 172.21.15.93:0/2970680691 learned_addr learned my addr 172.21.15.93:0/2970680691 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:25.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76ca7fc700 1 -- 172.21.15.93:0/2970680691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76cc0f55a0 msgr2=0x7f76cc105eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:25.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76ca7fc700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76cc0f55a0 0x7f76cc105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:25.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76ca7fc700 1 -- 172.21.15.93:0/2970680691 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc009640 con 0x7f76cc0f6840 2021-11-20T09:01:25.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76ca7fc700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f6840 0x7f76cc1063a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f76b400e970 tx=0x7f76b4007eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:25.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.973+0000 7f76c37fe700 1 -- 172.21.15.93:0/2970680691 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76b400c980 con 0x7f76cc0f6840 2021-11-20T09:01:25.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.974+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc05f4b0 con 0x7f76cc0f6840 2021-11-20T09:01:25.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.974+0000 7f76c37fe700 1 -- 172.21.15.93:0/2970680691 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76b4011470 con 0x7f76cc0f6840 2021-11-20T09:01:25.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.974+0000 7f76c37fe700 1 -- 172.21.15.93:0/2970680691 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76b4010630 con 0x7f76cc0f6840 2021-11-20T09:01:25.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.974+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc05f9f0 con 0x7f76cc0f6840 2021-11-20T09:01:25.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.974+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc046330 con 0x7f76cc0f6840 2021-11-20T09:01:25.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.975+0000 7f76c37fe700 1 -- 172.21.15.93:0/2970680691 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f76b40115e0 con 0x7f76cc0f6840 2021-11-20T09:01:25.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.975+0000 7f76c37fe700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f76b805c2a0 0x7f76b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:25.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.975+0000 7f76c37fe700 1 -- 172.21.15.93:0/2970680691 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f76b4079560 con 0x7f76cc0f6840 2021-11-20T09:01:25.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.975+0000 7f76caffd700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f76b805c2a0 0x7f76b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:25.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.976+0000 7f76caffd700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f76b805c2a0 0x7f76b805e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f76bc000f80 tx=0x7f76bc004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:25.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:25.979+0000 7f76c37fe700 1 -- 172.21.15.93:0/2970680691 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76b404bcc0 con 0x7f76cc0f6840 2021-11-20T09:01:26.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.164+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76cc0ffd70 con 0x7f76cc0f6840 2021-11-20T09:01:26.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.165+0000 7f76c37fe700 1 -- 172.21.15.93:0/2970680691 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f76b404f2e0 con 0x7f76cc0f6840 2021-11-20T09:01:26.164 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:26.164 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:26.165 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f76b805c2a0 msgr2=0x7f76b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:26.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f76b805c2a0 0x7f76b805e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f76bc000f80 tx=0x7f76bc004d10).stop 2021-11-20T09:01:26.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f6840 msgr2=0x7f76cc1063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:26.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f6840 0x7f76cc1063a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f76b400e970 tx=0x7f76b4007eb0).stop 2021-11-20T09:01:26.166 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 shutdown_connections 2021-11-20T09:01:26.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f76b805c2a0 0x7f76b805e760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:26.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76cc0f55a0 0x7f76cc105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:26.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 --2- 172.21.15.93:0/2970680691 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76cc0f6840 0x7f76cc1063a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:26.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 >> 172.21.15.93:0/2970680691 conn(0x7f76cc0f0ce0 msgr2=0x7f76cc0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:26.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 shutdown_connections 2021-11-20T09:01:26.168 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:26.166+0000 7f76d1289700 1 -- 172.21.15.93:0/2970680691 wait complete. 2021-11-20T09:01:26.168 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 96 2021-11-20T09:01:26.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:25 smithi102 conmon[36472]: audit 2021-11-20T09:01:24.831170+0000 mon.smithi093 (mon.0) 902 : audit [DBG] from='client.? 172.21.15.93:0/2536960872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:26.175 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:25 smithi093 conmon[33522]: audit 2021-11-20T09:01:24.831170+0000 mon.smithi093 (mon.0) 902 : audit [DBG] from='client.? 172.21.15.93:0/2536960872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:26.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:26.177 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:27.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:26 smithi102 conmon[36472]: cluster 2021-11-20T09:01:25.590412+0000 mgr.smithi093.qdikkm (mgr.14162) 326 : 2021-11-20T09:01:27.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:26 smithi102 conmon[36472]: cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 196 B/s wr, 1 op/s 2021-11-20T09:01:27.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:26 smithi102 conmon[36472]: audit 2021-11-20T09:01:26.165584+0000 mon.smithi093 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.93:0/2970680691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:27.179 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:27.206 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:26 smithi093 conmon[33522]: cluster 2021-11-20T09:01:25.590412+0000 mgr.smithi093.qdikkm (mgr.14162) 326 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 196 B/s wr, 1 op/s 2021-11-20T09:01:27.207 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:26 smithi093 conmon[33522]: audit 2021-11-20T09: 2021-11-20T09:01:27.207 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:26 smithi093 conmon[33522]: 01:26.165584+0000 mon.smithi093 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.93:0/2970680691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:27.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1438314422 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 msgr2=0x7fbacc0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:27.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1438314422 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc0f6c80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7fbabc004660 tx=0x7fbabc009b30).stop 2021-11-20T09:01:27.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1438314422 shutdown_connections 2021-11-20T09:01:27.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1438314422 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc0f6c80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:27.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1438314422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbacc0f5580 0x7fbacc0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:27.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1438314422 >> 172.21.15.93:0/1438314422 conn(0x7fbacc0f0ce0 msgr2=0x7fbacc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:27.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1438314422 shutdown_connections 2021-11-20T09:01:27.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.310+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1438314422 wait complete. 2021-11-20T09:01:27.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbad1a7a700 1 Processor -- start 2021-11-20T09:01:27.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbad1a7a700 1 -- start start 2021-11-20T09:01:27.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbad1a7a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbacc0f5580 0x7fbacc060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:27.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbad1a7a700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:27.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbad1a7a700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc107a10 con 0x7fbacc0f6820 2021-11-20T09:01:27.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbad1a7a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc10e520 con 0x7fbacc0f5580 2021-11-20T09:01:27.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbacaffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:27.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbacaffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc10e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44778/0 (socket says 172.21.15.93:44778) 2021-11-20T09:01:27.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.311+0000 7fbacaffd700 1 -- 172.21.15.93:0/1544987910 learned_addr learned my addr 172.21.15.93:0/1544987910 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:27.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbacb7fe700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbacc0f5580 0x7fbacc060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:27.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbacaffd700 1 -- 172.21.15.93:0/1544987910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbacc0f5580 msgr2=0x7fbacc060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:27.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbacaffd700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbacc0f5580 0x7fbacc060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:27.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbacaffd700 1 -- 172.21.15.93:0/1544987910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc005040 con 0x7fbacc0f6820 2021-11-20T09:01:27.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbacaffd700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc10e030 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fbabc009f50 tx=0x7fbabc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:27.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbab3fff700 1 -- 172.21.15.93:0/1544987910 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbabc018070 con 0x7fbacc0f6820 2021-11-20T09:01:27.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbab3fff700 1 -- 172.21.15.93:0/1544987910 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbabc013410 con 0x7fbacc0f6820 2021-11-20T09:01:27.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc10e660 con 0x7fbacc0f6820 2021-11-20T09:01:27.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbab3fff700 1 -- 172.21.15.93:0/1544987910 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbabc01c3c0 con 0x7fbacc0f6820 2021-11-20T09:01:27.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.312+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc10eb90 con 0x7fbacc0f6820 2021-11-20T09:01:27.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.313+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc027a00 con 0x7fbacc0f6820 2021-11-20T09:01:27.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.313+0000 7fbab3fff700 1 -- 172.21.15.93:0/1544987910 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbabc01d420 con 0x7fbacc0f6820 2021-11-20T09:01:27.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.314+0000 7fbab3fff700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbab405c2a0 0x7fbab405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:27.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.314+0000 7fbab3fff700 1 -- 172.21.15.93:0/1544987910 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fbabc078740 con 0x7fbacc0f6820 2021-11-20T09:01:27.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.314+0000 7fbacb7fe700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbab405c2a0 0x7fbab405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:27.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.314+0000 7fbacb7fe700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbab405c2a0 0x7fbab405e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbac0009880 tx=0x7fbac000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:27.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.317+0000 7fbab3fff700 1 -- 172.21.15.93:0/1544987910 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbabc04af90 con 0x7fbacc0f6820 2021-11-20T09:01:27.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.502+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbacc0fdd10 con 0x7fbacc0f6820 2021-11-20T09:01:27.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.502+0000 7fbab3fff700 1 -- 172.21.15.93:0/1544987910 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2191 (secure 0 0 0) 0x7fbabc04e5b0 con 0x7fbacc0f6820 2021-11-20T09:01:27.501 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:27.502 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:27.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbab405c2a0 msgr2=0x7fbab405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:27.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbab405c2a0 0x7fbab405e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbac0009880 tx=0x7fbac000b040).stop 2021-11-20T09:01:27.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 msgr2=0x7fbacc10e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:27.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc10e030 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fbabc009f50 tx=0x7fbabc009b30).stop 2021-11-20T09:01:27.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 shutdown_connections 2021-11-20T09:01:27.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fbab405c2a0 0x7fbab405e760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:27.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbacc0f5580 0x7fbacc060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:27.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 --2- 172.21.15.93:0/1544987910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbacc0f6820 0x7fbacc10e030 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:27.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 >> 172.21.15.93:0/1544987910 conn(0x7fbacc0f0ce0 msgr2=0x7fbacc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:27.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 shutdown_connections 2021-11-20T09:01:27.507 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:27.504+0000 7fbad1a7a700 1 -- 172.21.15.93:0/1544987910 wait complete. 2021-11-20T09:01:27.507 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 96 2021-11-20T09:01:27.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:27.515 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:28.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:27 smithi102 conmon[36472]: audit 2021-11-20T09:01:27.503289 2021-11-20T09:01:28.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:27 smithi102 conmon[36472]: +0000 mon.smithi093 (mon.0) 904 : audit [DBG] from='client.? 172.21.15.93:0/1544987910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:28.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:27 smithi093 conmon[33522]: audit 2021-11-20T09:01:27.503289+0000 mon.smithi093 (mon.0 2021-11-20T09:01:28.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:27 smithi093 conmon[33522]: ) 904 : audit [DBG] from='client.? 172.21.15.93:0/1544987910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:28.516 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:28.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 -- 172.21.15.93:0/2464385997 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f6840 msgr2=0x7ff4480f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:28.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/2464385997 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f6840 0x7ff4480f6ca0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7ff43c004660 tx=0x7ff43c009b30).stop 2021-11-20T09:01:28.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 -- 172.21.15.93:0/2464385997 shutdown_connections 2021-11-20T09:01:28.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/2464385997 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f6840 0x7ff4480f6ca0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:28.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/2464385997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4480f55a0 0x7ff4480f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:28.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 -- 172.21.15.93:0/2464385997 >> 172.21.15.93:0/2464385997 conn(0x7ff4480f0ce0 msgr2=0x7ff4480f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:28.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 -- 172.21.15.93:0/2464385997 shutdown_connections 2021-11-20T09:01:28.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 -- 172.21.15.93:0/2464385997 wait complete. 2021-11-20T09:01:28.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.642+0000 7ff44e9a9700 1 Processor -- start 2021-11-20T09:01:28.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff44e9a9700 1 -- start start 2021-11-20T09:01:28.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff44e9a9700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f55a0 0x7ff448105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:28.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff44e9a9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4480f6840 0x7ff448106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:28.648 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff44e9a9700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448106840 con 0x7ff4480f55a0 2021-11-20T09:01:28.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff44e9a9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448106980 con 0x7ff4480f6840 2021-11-20T09:01:28.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff447fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f55a0 0x7ff448105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:28.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff447fff700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f55a0 0x7ff448105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44786/0 (socket says 172.21.15.93:44786) 2021-11-20T09:01:28.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff447fff700 1 -- 172.21.15.93:0/735128588 learned_addr learned my addr 172.21.15.93:0/735128588 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:28.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff447fff700 1 -- 172.21.15.93:0/735128588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4480f6840 msgr2=0x7ff448106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:28.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff447fff700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4480f6840 0x7ff448106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:28.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff447fff700 1 -- 172.21.15.93:0/735128588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c005040 con 0x7ff4480f55a0 2021-11-20T09:01:28.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff447fff700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f55a0 0x7ff448105da0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7ff43800c530 tx=0x7ff438007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:28.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff444ff9700 1 -- 172.21.15.93:0/735128588 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff43800f650 con 0x7ff4480f55a0 2021-11-20T09:01:28.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff444ff9700 1 -- 172.21.15.93:0/735128588 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff43800fc60 con 0x7ff4480f55a0 2021-11-20T09:01:28.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44810b420 con 0x7ff4480f55a0 2021-11-20T09:01:28.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff444ff9700 1 -- 172.21.15.93:0/735128588 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff438012d20 con 0x7ff4480f55a0 2021-11-20T09:01:28.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.643+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44810b9e0 con 0x7ff4480f55a0 2021-11-20T09:01:28.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.644+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff448027a00 con 0x7ff4480f55a0 2021-11-20T09:01:28.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.645+0000 7ff444ff9700 1 -- 172.21.15.93:0/735128588 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff438019030 con 0x7ff4480f55a0 2021-11-20T09:01:28.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.645+0000 7ff444ff9700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff43005c2a0 0x7ff43005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:28.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.645+0000 7ff444ff9700 1 -- 172.21.15.93:0/735128588 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6758+0+0 (secure 0 0 0) 0x7ff438076040 con 0x7ff4480f55a0 2021-11-20T09:01:28.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.645+0000 7ff4477fe700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff43005c2a0 0x7ff43005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:28.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.645+0000 7ff4477fe700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff43005c2a0 0x7ff43005e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7ff43c009f50 tx=0x7ff43c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:28.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.649+0000 7ff444ff9700 1 -- 172.21.15.93:0/735128588 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4380462b0 con 0x7ff4480f55a0 2021-11-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.833+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4480fdd30 con 0x7ff4480f55a0 2021-11-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.833+0000 7ff444ff9700 1 -- 172.21.15.93:0/735128588 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2191 (secure 0 0 0) 0x7ff438017020 con 0x7ff4480f55a0 2021-11-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:18.801815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"cephfs.smithi102.hjodmi","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":8,"addr":"172.21.15.102:6827/2769927812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2769927812},{"type":"v1","addr":"172.21.15.102:6827","nonce":2769927812}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff43005c2a0 msgr2=0x7ff43005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff43005c2a0 0x7ff43005e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7ff43c009f50 tx=0x7ff43c009b30).stop 2021-11-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f55a0 msgr2=0x7ff448105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f55a0 0x7ff448105da0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7ff43800c530 tx=0x7ff438007e10).stop 2021-11-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 shutdown_connections 2021-11-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff43005c2a0 0x7ff43005e760 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4480f55a0 0x7ff448105da0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 --2- 172.21.15.93:0/735128588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4480f6840 0x7ff448106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 >> 172.21.15.93:0/735128588 conn(0x7ff4480f0ce0 msgr2=0x7ff4480fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 shutdown_connections 2021-11-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:28.835+0000 7ff44e9a9700 1 -- 172.21.15.93:0/735128588 wait complete. 2021-11-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 96 2021-11-20T09:01:28.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:18.801815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'cephfs.smithi102.hjodmi', 'rank': 0, 'incarnation': 92, 'state': 'up:clientreplay', 'state_seq': 8, 'addr': '172.21.15.102:6827/2769927812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6826', 'nonce': 2769927812}, {'type': 'v1', 'addr': '172.21.15.102:6827', 'nonce': 2769927812}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:28.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24593:up:clientreplay 2021-11-20T09:01:29.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:28 smithi102 conmon[36472]: cluster 2021-11-20T09:01:27.590790+0000 2021-11-20T09:01:29.126 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:28 smithi102 conmon[36472]: mgr.smithi093.qdikkm (mgr.14162) 327 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:01:29.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:28 smithi102 conmon[36472]: audit 2021-11-20T09:01:28.834323+0000 mon.smithi093 (mon. 2021-11-20T09:01:29.127 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:28 smithi102 conmon[36472]: 0) 905 : audit [DBG] from='client.? 172.21.15.93:0/735128588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:29.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:28 smithi093 conmon[33522]: cluster 2021-11-20T09:01:27.590790+0000 2021-11-20T09:01:29.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:28 smithi093 conmon[33522]: mgr.smithi093.qdikkm (mgr.14162) 327 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:01:29.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:28 smithi093 conmon[33522]: audit 2021-11-20T09:01:28.834323+0000 mon.smithi093 (mon.0) 905 : audit [DBG] from='client.? 172.21.15.93:0/735128588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:29.848 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:29.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:29 smithi093 conmon[33522]: debug 2021-11-20T09:01:29.836+0000 7f017bb78700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:01:29.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:29 smithi093 conmon[33522]: cluster 2021-11-20T09:01:29.415713 2021-11-20T09:01:29.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:29 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 906 : cluster [INF] daemon mds.cephfs.smithi102.hjodmi restarted 2021-11-20T09:01:29.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:29 smithi093 conmon[33522]: cluster 2021-11-20T09:01:29 2021-11-20T09:01:29.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:29 smithi093 conmon[33522]: .420170+0000 mon.smithi093 (mon.0) 907 : cluster [DBG] osdmap e60: 6 total, 6 up, 6 in 2021-11-20T09:01:29.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- 172.21.15.93:0/3912476520 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 msgr2=0x7f81a00f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:29.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 --2- 172.21.15.93:0/3912476520 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a00f59c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f819c004660 tx=0x7f819c009b30).stop 2021-11-20T09:01:29.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- 172.21.15.93:0/3912476520 shutdown_connections 2021-11-20T09:01:29.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 --2- 172.21.15.93:0/3912476520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81a00f6840 0x7f81a00f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:29.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 --2- 172.21.15.93:0/3912476520 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a00f59c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:29.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- 172.21.15.93:0/3912476520 >> 172.21.15.93:0/3912476520 conn(0x7f81a00f0ce0 msgr2=0x7f81a00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:29.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- 172.21.15.93:0/3912476520 shutdown_connections 2021-11-20T09:01:29.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- 172.21.15.93:0/3912476520 wait complete. 2021-11-20T09:01:29.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 Processor -- start 2021-11-20T09:01:29.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- start start 2021-11-20T09:01:29.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a0106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:29.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81a00f6840 0x7f81a0107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:29.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a01078e0 con 0x7f81a00f55a0 2021-11-20T09:01:29.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.976+0000 7f81a80ec700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a0107a20 con 0x7f81a00f6840 2021-11-20T09:01:29.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5e88700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a0106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:29.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5e88700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a0106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44796/0 (socket says 172.21.15.93:44796) 2021-11-20T09:01:29.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5e88700 1 -- 172.21.15.93:0/302843577 learned_addr learned my addr 172.21.15.93:0/302843577 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:29.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5687700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81a00f6840 0x7f81a0107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:29.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5e88700 1 -- 172.21.15.93:0/302843577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81a00f6840 msgr2=0x7f81a0107330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:29.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5e88700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81a00f6840 0x7f81a0107330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:29.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5e88700 1 -- 172.21.15.93:0/302843577 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c005040 con 0x7f81a00f55a0 2021-11-20T09:01:29.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a5e88700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a0106e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f819c004e30 tx=0x7f819c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:29.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81967fc700 1 -- 172.21.15.93:0/302843577 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f819c018070 con 0x7f81a00f55a0 2021-11-20T09:01:29.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81967fc700 1 -- 172.21.15.93:0/302843577 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f819c013410 con 0x7f81a00f55a0 2021-11-20T09:01:29.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a010c460 con 0x7f81a00f55a0 2021-11-20T09:01:29.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81967fc700 1 -- 172.21.15.93:0/302843577 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f819c01de20 con 0x7f81a00f55a0 2021-11-20T09:01:29.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.977+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a010ca20 con 0x7f81a00f55a0 2021-11-20T09:01:29.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.978+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a00fdd30 con 0x7f81a00f55a0 2021-11-20T09:01:29.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.978+0000 7f81967fc700 1 -- 172.21.15.93:0/302843577 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f819c01d420 con 0x7f81a00f55a0 2021-11-20T09:01:29.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.979+0000 7f81967fc700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f818c05c2a0 0x7f818c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:29.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.979+0000 7f81967fc700 1 -- 172.21.15.93:0/302843577 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f819c078020 con 0x7f81a00f55a0 2021-11-20T09:01:29.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.979+0000 7f81a5687700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f818c05c2a0 0x7f818c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:29.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.979+0000 7f81a5687700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f818c05c2a0 0x7f818c05e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f8190004ce0 tx=0x7f819000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:29.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:29.983+0000 7f81967fc700 1 -- 172.21.15.93:0/302843577 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f819c04a7b0 con 0x7f81a00f55a0 2021-11-20T09:01:30.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.168+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81a010d170 con 0x7f81a00f55a0 2021-11-20T09:01:30.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:29 smithi102 conmon[36472]: cluster 2021-11-20T09:01:29.415713+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:30.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:29 smithi102 conmon[36472]: 906 : cluster [INF] daemon mds.cephfs.smithi102.hjodmi restarted 2021-11-20T09:01:30.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:29 smithi102 conmon[36472]: cluster 2021-11-20T09:01:29.420170+0000 mon.smithi093 2021-11-20T09:01:30.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:29 smithi102 conmon[36472]: (mon.0) 907 : cluster [DBG] osdmap e60: 6 total, 6 up, 6 in 2021-11-20T09:01:30.169 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.169+0000 7f81967fc700 1 -- 172.21.15.93:0/302843577 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2186 (secure 0 0 0) 0x7f819c04ddd0 con 0x7f81a00f55a0 2021-11-20T09:01:30.170 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:30.170 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:29.847749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:replay","state_seq":2,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:30.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f818c05c2a0 msgr2=0x7f818c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:30.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f818c05c2a0 0x7f818c05e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f8190004ce0 tx=0x7f819000b040).stop 2021-11-20T09:01:30.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 msgr2=0x7f81a0106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:30.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a0106e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f819c004e30 tx=0x7f819c009b30).stop 2021-11-20T09:01:30.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 shutdown_connections 2021-11-20T09:01:30.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f818c05c2a0 0x7f818c05e760 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:30.172 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f81a00f55a0 0x7f81a0106e40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:30.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 --2- 172.21.15.93:0/302843577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81a00f6840 0x7f81a0107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:30.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.170+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 >> 172.21.15.93:0/302843577 conn(0x7f81a00f0ce0 msgr2=0x7f81a005f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:30.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.171+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 shutdown_connections 2021-11-20T09:01:30.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:30.171+0000 7f81a80ec700 1 -- 172.21.15.93:0/302843577 wait complete. 2021-11-20T09:01:30.174 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 98 2021-11-20T09:01:30.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:29.847749+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:30.181 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:replay 2021-11-20T09:01:31.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster 2021-11-20T09:01:29.591222+0000 mgr.smithi093.qdikkm (mgr.14162) 328 : cluster [DBG] 2021-11-20T09:01:31.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: pgmap v308: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 204 B/s wr, 0 op/s 2021-11-20T09:01:31.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster 2021-11-20T09:01:29.837192+0000 mon.smithi093 (mon.0) 908 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:01:31.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster 2021-11-20T09:01:29.847632+0000 mon.smithi093 (mon.0) 909 : cluster 2021-11-20T09:01:31.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: [DBG] mds.? [v2:172.21.15.102:6826/3890204616,v1:172.21.15.102:6827/3890204616] up:boot 2021-11-20T09:01:31.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster 2021-11-20T09:01:29.847711+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:31.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: 910 : cluster [INF] Standby daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 2021-11-20T09:01:31.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster 2021-11-20T09:01: 2021-11-20T09:01:31.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: 29.847798+0000 mon.smithi093 (mon.0) 911 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:01:31.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster 2021-11-20 2021-11-20T09:01:31.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: T09:01:29.848174+0000 mon.smithi093 (mon.0) 912 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:01:31.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: audit 2021-11- 2021-11-20T09:01:31.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: 20T09:01:29.848288+0000 mon.smithi093 (mon.0) 913 2021-11-20T09:01:31.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T09:01:31.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster 2021-11 2021-11-20T09:01:31.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: -20T09:01:29.852612+0000 mon.smithi093 (mon.0) 914 : 2021-11-20T09:01:31.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:replay} 1 up:standby 2021-11-20T09:01:31.171 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: audit 2021-11 2021-11-20T09:01:31.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: -20T09:01:30.169934+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:31.172 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:30 smithi102 conmon[36472]: 915 : audit [DBG] from='client.? 172.21.15.93:0/302843577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:31.183 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:31.209 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster 2021-11-20T09:01:29.591222+0000 mgr.smithi093.qdikkm (mgr.14162) 328 : 2021-11-20T09:01:31.209 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster [DBG] pgmap v308: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 204 B/s wr, 0 op/s 2021-11-20T09:01:31.210 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster 2021-11-20T09:01:29.837192+0000 mon.smithi093 (mon.0) 908 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T09:01:31.210 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster 2021-11-20T09:01:29.847632+0000 mon.smithi093 (mon.0) 909 2021-11-20T09:01:31.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: : cluster [DBG] mds.? [v2:172.21.15.102:6826/3890204616,v1:172.21.15.102:6827/3890204616] up:boot 2021-11-20T09:01:31.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster 2021-11-20T09:01:29. 2021-11-20T09:01:31.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: 847711+0000 mon.smithi093 (mon.0) 910 : cluster 2021-11-20T09:01:31.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: [INF] Standby daemon mds.cephfs.smithi102.lhrlmh assigned to filesystem cephfs as rank 0 2021-11-20T09:01:31.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster 2021-11-20T09:01:31.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: 2021-11-20T09:01:29.847798+0000 mon.smithi093 (mon 2021-11-20T09:01:31.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: .0) 911 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T09:01:31.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster 2021-11 2021-11-20T09:01:31.214 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: -20T09: 2021-11-20T09:01:31.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: 01:29.848174 2021-11-20T09:01:31.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 912 : cluster 2021-11-20T09:01:31.215 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T09:01:31.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: audit 2021-11-20T 2021-11-20T09:01:31.216 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: 09:01:29.848288+0000 mon.smithi093 (mon. 2021-11-20T09:01:31.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: 0) 913 : audit [DBG] from='mgr.14162 172.21.15.93:0/4151049494' entity='mgr.smithi093.qdikkm' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi102.hjodmi"}]: dispatch 2021-11-20T09:01:31.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T09:01:31.217 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: T09:01:29.852612+0000 mon.smithi093 ( 2021-11-20T09:01:31.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: mon.0) 914 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:replay} 1 up:standby 2021-11-20T09:01:31.218 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: audit 2021 2021-11-20T09:01:31.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: -11-20T09:01:30.169934 2021-11-20T09:01:31.219 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: +0000 mon.smithi093 (mon.0) 915 : audit [DBG] 2021-11-20T09:01:31.220 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:30 smithi093 conmon[33522]: from='client.? 172.21.15.93:0/302843577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:31.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 -- 172.21.15.93:0/4257226265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a00f9730 msgr2=0x7f47a00fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:31.305 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 --2- 172.21.15.93:0/4257226265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a00f9730 0x7f47a00fbb80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f4794004660 tx=0x7f4794009b30).stop 2021-11-20T09:01:31.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 -- 172.21.15.93:0/4257226265 shutdown_connections 2021-11-20T09:01:31.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 --2- 172.21.15.93:0/4257226265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a00f9730 0x7f47a00fbb80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:31.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 --2- 172.21.15.93:0/4257226265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a0060430 0x7f47a00f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:31.306 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 -- 172.21.15.93:0/4257226265 >> 172.21.15.93:0/4257226265 conn(0x7f47a00f0c10 msgr2=0x7f47a00f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:31.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 -- 172.21.15.93:0/4257226265 shutdown_connections 2021-11-20T09:01:31.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.304+0000 7f47a485d700 1 -- 172.21.15.93:0/4257226265 wait complete. 2021-11-20T09:01:31.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f47a485d700 1 Processor -- start 2021-11-20T09:01:31.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f47a485d700 1 -- start start 2021-11-20T09:01:31.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f47a485d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a0060430 0x7f47a0106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:31.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f47a485d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a00f9730 0x7f47a01072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:31.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f47a485d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0107940 con 0x7f47a0060430 2021-11-20T09:01:31.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f47a485d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01080f0 con 0x7f47a00f9730 2021-11-20T09:01:31.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f479f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a0060430 0x7f47a0106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:31.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f479f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a0060430 0x7f47a0106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44806/0 (socket says 172.21.15.93:44806) 2021-11-20T09:01:31.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f479f7fe700 1 -- 172.21.15.93:0/2766824547 learned_addr learned my addr 172.21.15.93:0/2766824547 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:31.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.305+0000 7f479effd700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a00f9730 0x7f47a01072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:31.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f479f7fe700 1 -- 172.21.15.93:0/2766824547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a00f9730 msgr2=0x7f47a01072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:31.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f479f7fe700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a00f9730 0x7f47a01072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:31.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f479f7fe700 1 -- 172.21.15.93:0/2766824547 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4794005040 con 0x7f47a0060430 2021-11-20T09:01:31.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f479f7fe700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a0060430 0x7f47a0106dc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f479000b410 tx=0x7f4790006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:31.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f4787fff700 1 -- 172.21.15.93:0/2766824547 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47900125f0 con 0x7f47a0060430 2021-11-20T09:01:31.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f4787fff700 1 -- 172.21.15.93:0/2766824547 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f479000be30 con 0x7f47a0060430 2021-11-20T09:01:31.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a01083d0 con 0x7f47a0060430 2021-11-20T09:01:31.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f4787fff700 1 -- 172.21.15.93:0/2766824547 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4790002a20 con 0x7f47a0060430 2021-11-20T09:01:31.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.306+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a0108990 con 0x7f47a0060430 2021-11-20T09:01:31.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.307+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a00f48c0 con 0x7f47a0060430 2021-11-20T09:01:31.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.307+0000 7f4787fff700 1 -- 172.21.15.93:0/2766824547 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f479000f030 con 0x7f47a0060430 2021-11-20T09:01:31.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.307+0000 7f4787fff700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f478805c300 0x7f478805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:31.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.308+0000 7f4787fff700 1 -- 172.21.15.93:0/2766824547 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f47900768b0 con 0x7f47a0060430 2021-11-20T09:01:31.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.308+0000 7f479effd700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f478805c300 0x7f478805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:31.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.308+0000 7f479effd700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f478805c300 0x7f478805e7c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f47a0106b50 tx=0x7f4794009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:31.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.311+0000 7f4787fff700 1 -- 172.21.15.93:0/2766824547 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4790048fc0 con 0x7f47a0060430 2021-11-20T09:01:31.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.496+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47a0045ad0 con 0x7f47a0060430 2021-11-20T09:01:31.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.497+0000 7f4787fff700 1 -- 172.21.15.93:0/2766824547 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+2189 (secure 0 0 0) 0x7f479004c5e0 con 0x7f47a0060430 2021-11-20T09:01:31.497 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:31.498 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:30.850906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:31.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f478805c300 msgr2=0x7f478805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:31.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f478805c300 0x7f478805e7c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f47a0106b50 tx=0x7f4794009b30).stop 2021-11-20T09:01:31.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a0060430 msgr2=0x7f47a0106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:31.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a0060430 0x7f47a0106dc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f479000b410 tx=0x7f4790006ef0).stop 2021-11-20T09:01:31.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 shutdown_connections 2021-11-20T09:01:31.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f478805c300 0x7f478805e7c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:31.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47a0060430 0x7f47a0106dc0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:31.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 --2- 172.21.15.93:0/2766824547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47a00f9730 0x7f47a01072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:31.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.498+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 >> 172.21.15.93:0/2766824547 conn(0x7f47a00f0c10 msgr2=0x7f47a00f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:31.506 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.499+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 shutdown_connections 2021-11-20T09:01:31.507 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:31.499+0000 7f47a485d700 1 -- 172.21.15.93:0/2766824547 wait complete. 2021-11-20T09:01:31.507 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 99 2021-11-20T09:01:31.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:30.850906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:reconnect', 'state_seq': 5, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:31.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:reconnect 2021-11-20T09:01:32.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:31 smithi102 conmon[36472]: cluster 2021-11-20T09:01:30.853624+0000 mon.smithi093 (mon.0) 916 : cluster 2021-11-20T09:01:32.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:31 smithi102 conmon[36472]: [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:reconnect 2021-11-20T09:01:32.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:31 smithi102 conmon[36472]: cluster 2021-11-20T09:01:30.853721+0000 mon.smithi093 (mon.0) 917 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:reconnect} 1 up:standby 2021-11-20T09:01:32.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:31 smithi102 conmon[36472]: cluster 2021-11-20T09:01:30.873743+0000 mds.cephfs.smithi102.lhrlmh (mds. 2021-11-20T09:01:32.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:31 smithi102 conmon[36472]: 0) 1 : cluster [DBG] reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:01:32.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:31 smithi102 conmon[36472]: audit 2021-11 2021-11-20T09:01:32.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:31 smithi102 conmon[36472]: -20T09:01:31.497946+0000 mon.smithi093 (mon.0) 918 : audit [DBG] from='client.? 172.21.15.93:0/2766824547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:32.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: cluster 2021-11-20T09:01:30.853624+0000 mon.smithi093 (mon. 2021-11-20T09:01:32.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: 0) 916 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:reconnect 2021-11-20T09:01:32.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: cluster 2021-11-20T09:01:30.853721+0000 mon.smithi093 (mon.0) 917 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:reconnect} 1 up:standby 2021-11-20T09:01:32.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: cluster 2021-11-20T09:01:32.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: 2021-11-20T09:01:30.873743 2021-11-20T09:01:32.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: +0000 mds.cephfs.smithi102.lhrlmh (mds.0) 1 : cluster [DBG] 2021-11-20T09:01:32.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: reconnect by client.14504 v1:192.168.0.1:0/693079086 after 0.0209996 2021-11-20T09:01:32.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: audit 2021-11 2021-11-20T09:01:32.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: -20T09:01:31.497946+0000 mon.smithi093 (mon. 2021-11-20T09:01:32.213 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:31 smithi093 conmon[33522]: 0) 918 : audit [DBG] from='client.? 172.21.15.93:0/2766824547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:32.510 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:32.636 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 -- 172.21.15.93:0/648379869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f6820 msgr2=0x7fe9300f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:32.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 --2- 172.21.15.93:0/648379869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f6820 0x7fe9300f6c80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fe924004660 tx=0x7fe924009b30).stop 2021-11-20T09:01:32.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 -- 172.21.15.93:0/648379869 shutdown_connections 2021-11-20T09:01:32.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 --2- 172.21.15.93:0/648379869 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f6820 0x7fe9300f6c80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:32.637 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 --2- 172.21.15.93:0/648379869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9300f5580 0x7fe9300f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:32.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 -- 172.21.15.93:0/648379869 >> 172.21.15.93:0/648379869 conn(0x7fe9300f0ce0 msgr2=0x7fe9300f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:32.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 -- 172.21.15.93:0/648379869 shutdown_connections 2021-11-20T09:01:32.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.635+0000 7fe934e50700 1 -- 172.21.15.93:0/648379869 wait complete. 2021-11-20T09:01:32.638 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 Processor -- start 2021-11-20T09:01:32.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 -- start start 2021-11-20T09:01:32.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f5580 0x7fe930105d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:32.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9300f6820 0x7fe930106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:32.639 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9301067c0 con 0x7fe9300f5580 2021-11-20T09:01:32.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930106900 con 0x7fe9300f6820 2021-11-20T09:01:32.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f5580 0x7fe930105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:32.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92e59c700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f5580 0x7fe930105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44816/0 (socket says 172.21.15.93:44816) 2021-11-20T09:01:32.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92e59c700 1 -- 172.21.15.93:0/43101151 learned_addr learned my addr 172.21.15.93:0/43101151 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:32.640 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92dd9b700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9300f6820 0x7fe930106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:32.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92e59c700 1 -- 172.21.15.93:0/43101151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9300f6820 msgr2=0x7fe930106210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:32.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92e59c700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9300f6820 0x7fe930106210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:32.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92e59c700 1 -- 172.21.15.93:0/43101151 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe924005040 con 0x7fe9300f5580 2021-11-20T09:01:32.641 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe92e59c700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f5580 0x7fe930105d20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fe92000eb70 tx=0x7fe92000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:32.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe91effd700 1 -- 172.21.15.93:0/43101151 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9200092e0 con 0x7fe9300f5580 2021-11-20T09:01:32.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93010b3a0 con 0x7fe9300f5580 2021-11-20T09:01:32.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe91effd700 1 -- 172.21.15.93:0/43101151 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe92000f040 con 0x7fe9300f5580 2021-11-20T09:01:32.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe91effd700 1 -- 172.21.15.93:0/43101151 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92000c8a0 con 0x7fe9300f5580 2021-11-20T09:01:32.642 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.636+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93010b960 con 0x7fe9300f5580 2021-11-20T09:01:32.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.637+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe930046330 con 0x7fe9300f5580 2021-11-20T09:01:32.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.638+0000 7fe91effd700 1 -- 172.21.15.93:0/43101151 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe92001b030 con 0x7fe9300f5580 2021-11-20T09:01:32.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.638+0000 7fe91effd700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe91805c2a0 0x7fe91805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:32.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.638+0000 7fe91effd700 1 -- 172.21.15.93:0/43101151 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fe920076410 con 0x7fe9300f5580 2021-11-20T09:01:32.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.638+0000 7fe92dd9b700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe91805c2a0 0x7fe91805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:32.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.638+0000 7fe92dd9b700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe91805c2a0 0x7fe91805e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fe924005010 tx=0x7fe924016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:32.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.642+0000 7fe91effd700 1 -- 172.21.15.93:0/43101151 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe920048bd0 con 0x7fe9300f5580 2021-11-20T09:01:32.823 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.825+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe930027a00 con 0x7fe9300f5580 2021-11-20T09:01:32.824 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.825+0000 7fe91effd700 1 -- 172.21.15.93:0/43101151 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2188 (secure 0 0 0) 0x7fe9200198a0 con 0x7fe9300f5580 2021-11-20T09:01:32.825 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:32.825 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:31.852286+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:32.827 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe91805c2a0 msgr2=0x7fe91805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:32.827 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe91805c2a0 0x7fe91805e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fe924005010 tx=0x7fe924016040).stop 2021-11-20T09:01:32.827 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f5580 msgr2=0x7fe930105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:32.827 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f5580 0x7fe930105d20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fe92000eb70 tx=0x7fe92000b040).stop 2021-11-20T09:01:32.828 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 shutdown_connections 2021-11-20T09:01:32.828 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fe91805c2a0 0x7fe91805e760 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:32.828 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9300f5580 0x7fe930105d20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:32.828 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 --2- 172.21.15.93:0/43101151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9300f6820 0x7fe930106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:32.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 >> 172.21.15.93:0/43101151 conn(0x7fe9300f0ce0 msgr2=0x7fe9300fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:32.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 shutdown_connections 2021-11-20T09:01:32.829 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:32.827+0000 7fe934e50700 1 -- 172.21.15.93:0/43101151 wait complete. 2021-11-20T09:01:32.829 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 100 2021-11-20T09:01:32.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:31.852286+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:32.839 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:rejoin 2021-11-20T09:01:33.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: cluster 2021-11-20T09:01:31.591667+0000 mgr.smithi093.qdikkm (mgr.14162) 329 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T09:01:33.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: cluster 2021-11-20T09:01:31.857237+0000 mon.smithi093 (mon.0) 919 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:rejoin 2021-11-20T09:01:33.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: cluster 2021-11-20T09:01:33.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: 2021-11-20T09:01:31.857285+0000 mon.smithi093 2021-11-20T09:01:33.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: (mon.0) 920 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:rejoin} 1 up:standby 2021-11-20T09:01:33.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: audit 2021-11-20T09:01:33.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: 2021-11-20T09:01: 2021-11-20T09:01:33.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: 32.826264+0000 mon.smithi093 (mon.0) 921 : 2021-11-20T09:01:33.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:32 smithi102 conmon[36472]: audit [DBG] from='client.? 172.21.15.93:0/43101151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:33.189 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: cluster 2021-11-20T09:01:31.591667+0000 mgr.smithi093.qdikkm (mgr.14162) 329 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T09:01:33.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: cluster 2021-11-20T09:01:31.857237+0000 mon.smithi093 (mon.0) 919 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:rejoin 2021-11-20T09:01:33.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: cluster 2021-11 2021-11-20T09:01:33.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: -20T09:01:31.857285+0000 mon.smithi093 (mon.0) 920 : 2021-11-20T09:01:33.191 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:rejoin} 1 up:standby 2021-11-20T09:01:33.191 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: audit 2021-11-20T09:01 2021-11-20T09:01:33.191 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: :32.826264+0000 mon.smithi093 (mon.0) 2021-11-20T09:01:33.191 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:32 smithi093 conmon[33522]: 921 : audit [DBG] from='client.? 172.21.15.93:0/43101151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:33.840 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:33.960 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 -- 172.21.15.93:0/533633072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 msgr2=0x7f6c000f7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:33.960 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/533633072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c000f7d70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f6bf4004660 tx=0x7f6bf4009b30).stop 2021-11-20T09:01:33.960 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 -- 172.21.15.93:0/533633072 shutdown_connections 2021-11-20T09:01:33.961 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/533633072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c000f7d70 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:33.961 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/533633072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c000f6670 0x7f6c000f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:33.961 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 -- 172.21.15.93:0/533633072 >> 172.21.15.93:0/533633072 conn(0x7f6c000f1db0 msgr2=0x7f6c000f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:33.962 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 -- 172.21.15.93:0/533633072 shutdown_connections 2021-11-20T09:01:33.962 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 -- 172.21.15.93:0/533633072 wait complete. 2021-11-20T09:01:33.963 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 Processor -- start 2021-11-20T09:01:33.963 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.958+0000 7f6c04ae3700 1 -- start start 2021-11-20T09:01:33.963 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6c04ae3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c000f6670 0x7f6c00106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:33.963 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6c04ae3700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c00107370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:33.964 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6c04ae3700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00107920 con 0x7f6c000f7910 2021-11-20T09:01:33.964 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6c04ae3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00107a60 con 0x7f6c000f6670 2021-11-20T09:01:33.964 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6bfdd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c00107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:33.964 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6bfdd9b700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c00107370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44830/0 (socket says 172.21.15.93:44830) 2021-11-20T09:01:33.965 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6bfdd9b700 1 -- 172.21.15.93:0/1651100177 learned_addr learned my addr 172.21.15.93:0/1651100177 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:33.965 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6bfdd9b700 1 -- 172.21.15.93:0/1651100177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c000f6670 msgr2=0x7f6c00106e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:33.965 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6bfdd9b700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c000f6670 0x7f6c00106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:33.965 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6bfdd9b700 1 -- 172.21.15.93:0/1651100177 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf4005040 con 0x7f6c000f7910 2021-11-20T09:01:33.965 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6bfdd9b700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c00107370 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f6bf4009830 tx=0x7f6bf4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:33.966 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6beeffd700 1 -- 172.21.15.93:0/1651100177 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bf4018070 con 0x7f6c000f7910 2021-11-20T09:01:33.966 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6beeffd700 1 -- 172.21.15.93:0/1651100177 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bf4013410 con 0x7f6c000f7910 2021-11-20T09:01:33.966 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0010c4a0 con 0x7f6c000f7910 2021-11-20T09:01:33.966 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6beeffd700 1 -- 172.21.15.93:0/1651100177 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bf401c620 con 0x7f6c000f7910 2021-11-20T09:01:33.967 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.959+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0010ca60 con 0x7f6c000f7910 2021-11-20T09:01:33.967 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.960+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c00027a00 con 0x7f6c000f7910 2021-11-20T09:01:33.967 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.961+0000 7f6beeffd700 1 -- 172.21.15.93:0/1651100177 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6bf401c8e0 con 0x7f6c000f7910 2021-11-20T09:01:33.967 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.961+0000 7f6beeffd700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6be805c2a0 0x7f6be805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:33.967 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.961+0000 7f6beeffd700 1 -- 172.21.15.93:0/1651100177 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f6bf4078350 con 0x7f6c000f7910 2021-11-20T09:01:33.968 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.961+0000 7f6bfe59c700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6be805c2a0 0x7f6be805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:33.968 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.961+0000 7f6bfe59c700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6be805c2a0 0x7f6be805e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f6c000f7770 tx=0x7f6bf80079e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:33.969 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:33.965+0000 7f6beeffd700 1 -- 172.21.15.93:0/1651100177 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bf404aa00 con 0x7f6c000f7910 2021-11-20T09:01:34.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.159+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c0010c630 con 0x7f6c000f7910 2021-11-20T09:01:34.158 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.159+0000 7f6beeffd700 1 -- 172.21.15.93:0/1651100177 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7f6c0010c630 con 0x7f6c000f7910 2021-11-20T09:01:34.159 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:34.159 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:34.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6be805c2a0 msgr2=0x7f6be805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:34.160 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6be805c2a0 0x7f6be805e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f6c000f7770 tx=0x7f6bf80079e0).stop 2021-11-20T09:01:34.161 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 msgr2=0x7f6c00107370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:34.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c00107370 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f6bf4009830 tx=0x7f6bf4009b30).stop 2021-11-20T09:01:34.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 shutdown_connections 2021-11-20T09:01:34.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f6be805c2a0 0x7f6be805e760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:34.162 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c000f6670 0x7f6c00106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:34.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 --2- 172.21.15.93:0/1651100177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c000f7910 0x7f6c00107370 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:34.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 >> 172.21.15.93:0/1651100177 conn(0x7f6c000f1db0 msgr2=0x7f6c000fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:34.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 shutdown_connections 2021-11-20T09:01:34.163 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:34.161+0000 7f6c04ae3700 1 -- 172.21.15.93:0/1651100177 wait complete. 2021-11-20T09:01:34.164 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:34.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:33 smithi102 conmon[36472]: cluster 2021-11-20T09:01:32.858998+0000 mon.smithi093 (mon.0) 922 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:clientreplay 2021-11-20T09:01:34.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:33 smithi102 conmon[36472]: cluster 2021-11-20T09:01:34.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:33 smithi102 conmon[36472]: 2021-11-20T09:01:32.859055+0000 mon.smithi093 (mon.0) 923 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:34.170 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:33 smithi093 conmon[33522]: cluster 2021-11-20T09:01:32.858998+0000 2021-11-20T09:01:34.170 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:33 smithi093 conmon[33522]: mon.smithi093 (mon.0) 922 : cluster [DBG] mds.? [v2:172.21.15.102:6824/3315610363,v1:172.21.15.102:6825/3315610363] up:clientreplay 2021-11-20T09:01:34.170 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:33 smithi093 conmon[33522]: cluster 2021-11-20 2021-11-20T09:01:34.171 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:33 smithi093 conmon[33522]: T09:01:32.859055+0000 mon.smithi093 (mon.0) 923 : cluster [DBG] 2021-11-20T09:01:34.171 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:33 smithi093 conmon[33522]: fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:34.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:34.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:35.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:34 smithi102 conmon[36472]: cluster 2021-11-20T09:01:33.592018+0000 mgr.smithi093.qdikkm (mgr.14162) 330 : cluster [DBG] pgmap v310: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T09:01:35.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:34 smithi102 conmon[36472]: cluster 2021-11-20T09:01:33.869487+0000 mon.smithi093 (mon.0) 924 : cluster [DBG] mds.? [v2:172.21.15.102:6826/3890204616,v1:172.21.15.102:6827/3890204616] up:standby 2021-11-20T09:01:35.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:34 smithi102 conmon[36472]: cluster 2021-11-20T09:01:35.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:34 smithi102 conmon[36472]: 2021-11-20T09:01:33.869535+0000 mon.smithi093 (mon.0) 925 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:35.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:34 smithi102 conmon[36472]: audit 2021-11-20T09:01:34. 2021-11-20T09:01:35.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:34 smithi102 conmon[36472]: 160097+0000 mon.smithi093 (mon.0) 926 : audit [DBG] from='client.? 172.21.15.93:0/1651100177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:35.173 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:35.201 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: cluster 2021-11-20T09:01:33.592018+0000 mgr.smithi093.qdikkm (mgr.14162) 330 : cluster 2021-11-20T09:01:35.202 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: [DBG] pgmap v310: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T09:01:35.202 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: cluster 2021-11-20T09:01:33.869487+0000 mon.smithi093 (mon.0) 924 : cluster [DBG] mds.? [v2:172.21.15.102:6826/3890204616,v1:172.21.15.102:6827/3890204616] up:standby 2021-11-20T09:01:35.202 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: cluster 2021-11-20T09:01 2021-11-20T09:01:35.203 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: :33.869535+0000 mon.smithi093 (mon.0) 925 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi102.lhrlmh=up:clientreplay} 1 up:standby 2021-11-20T09:01:35.204 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: audit 2021-11-20 2021-11-20T09:01:35.204 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: T09:01:34.160097+0000 mon.smithi093 (mon.0) 926 : audit [DBG] 2021-11-20T09:01:35.204 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:34 smithi093 conmon[33522]: from='client.? 172.21.15.93:0/1651100177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:35.307 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 -- 172.21.15.93:0/373921489 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 msgr2=0x7f4c680f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:35.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 --2- 172.21.15.93:0/373921489 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c680f7da0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f4c5c004660 tx=0x7f4c5c009b30).stop 2021-11-20T09:01:35.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 -- 172.21.15.93:0/373921489 shutdown_connections 2021-11-20T09:01:35.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 --2- 172.21.15.93:0/373921489 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c680f7da0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:35.308 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 --2- 172.21.15.93:0/373921489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c680f66a0 0x7f4c680f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:35.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 -- 172.21.15.93:0/373921489 >> 172.21.15.93:0/373921489 conn(0x7f4c680f1e00 msgr2=0x7f4c680f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:35.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 -- 172.21.15.93:0/373921489 shutdown_connections 2021-11-20T09:01:35.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.306+0000 7f4c6e847700 1 -- 172.21.15.93:0/373921489 wait complete. 2021-11-20T09:01:35.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c6e847700 1 Processor -- start 2021-11-20T09:01:35.309 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c6e847700 1 -- start start 2021-11-20T09:01:35.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c6e847700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c680f66a0 0x7f4c68106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:35.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c6e847700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c68107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:35.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c6e847700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68107900 con 0x7f4c680f7940 2021-11-20T09:01:35.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c6e847700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68107a40 con 0x7f4c680f66a0 2021-11-20T09:01:35.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c677fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c68107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:35.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c677fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c68107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44844/0 (socket says 172.21.15.93:44844) 2021-11-20T09:01:35.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.307+0000 7f4c677fe700 1 -- 172.21.15.93:0/1417942498 learned_addr learned my addr 172.21.15.93:0/1417942498 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:35.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c67fff700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c680f66a0 0x7f4c68106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:35.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c677fe700 1 -- 172.21.15.93:0/1417942498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c680f66a0 msgr2=0x7f4c68106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:35.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c677fe700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c680f66a0 0x7f4c68106e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:35.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c677fe700 1 -- 172.21.15.93:0/1417942498 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c005040 con 0x7f4c680f7940 2021-11-20T09:01:35.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c677fe700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c68107350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f4c5c009f50 tx=0x7f4c5c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:35.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c64ff9700 1 -- 172.21.15.93:0/1417942498 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c018070 con 0x7f4c680f7940 2021-11-20T09:01:35.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6810c480 con 0x7f4c680f7940 2021-11-20T09:01:35.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c64ff9700 1 -- 172.21.15.93:0/1417942498 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c5c013410 con 0x7f4c680f7940 2021-11-20T09:01:35.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c64ff9700 1 -- 172.21.15.93:0/1417942498 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c01c3c0 con 0x7f4c680f7940 2021-11-20T09:01:35.313 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.308+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6810ca40 con 0x7f4c680f7940 2021-11-20T09:01:35.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.309+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c68027a00 con 0x7f4c680f7940 2021-11-20T09:01:35.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.309+0000 7f4c64ff9700 1 -- 172.21.15.93:0/1417942498 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4c5c01d420 con 0x7f4c680f7940 2021-11-20T09:01:35.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.310+0000 7f4c64ff9700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c5005c2a0 0x7f4c5005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:35.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.310+0000 7f4c64ff9700 1 -- 172.21.15.93:0/1417942498 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f4c5c078170 con 0x7f4c680f7940 2021-11-20T09:01:35.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.310+0000 7f4c67fff700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c5005c2a0 0x7f4c5005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:35.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.310+0000 7f4c67fff700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c5005c2a0 0x7f4c5005e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f4c680f77a0 tx=0x7f4c58009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:35.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.313+0000 7f4c64ff9700 1 -- 172.21.15.93:0/1417942498 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c5c04a880 con 0x7f4c680f7940 2021-11-20T09:01:35.495 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.496+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c680fee30 con 0x7f4c680f7940 2021-11-20T09:01:35.496 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.497+0000 7f4c64ff9700 1 -- 172.21.15.93:0/1417942498 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7f4c5c04dea0 con 0x7f4c680f7940 2021-11-20T09:01:35.497 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:35.498 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:35.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c5005c2a0 msgr2=0x7f4c5005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:35.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c5005c2a0 0x7f4c5005e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f4c680f77a0 tx=0x7f4c58009c00).stop 2021-11-20T09:01:35.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 msgr2=0x7f4c68107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:35.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c68107350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f4c5c009f50 tx=0x7f4c5c009b30).stop 2021-11-20T09:01:35.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 shutdown_connections 2021-11-20T09:01:35.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f4c5005c2a0 0x7f4c5005e760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:35.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c680f66a0 0x7f4c68106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:35.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 --2- 172.21.15.93:0/1417942498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c680f7940 0x7f4c68107350 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:35.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.498+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 >> 172.21.15.93:0/1417942498 conn(0x7f4c680f1e00 msgr2=0x7f4c680fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:35.501 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.499+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 shutdown_connections 2021-11-20T09:01:35.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:35.499+0000 7f4c6e847700 1 -- 172.21.15.93:0/1417942498 wait complete. 2021-11-20T09:01:35.502 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:35.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:35.510 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:36.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:35 smithi102 conmon[36472]: audit 2021-11-20T09:01:35.497850+0000 mon.smithi093 (mon.0 2021-11-20T09:01:36.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:35 smithi102 conmon[36472]: ) 927 : audit [DBG] from='client.? 172.21.15.93:0/1417942498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:36.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:35 smithi093 conmon[33522]: audit 2021-11-20T09:01:35.497850+0000 mon.smithi093 (mon.0) 927 : audit [DBG] from='client.? 172.21.15.93:0/1417942498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:36.511 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:36.643 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2904707214 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f4840 msgr2=0x7f78ac0f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:36.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2904707214 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f4840 0x7f78ac0f4c60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f789c004660 tx=0x7f789c009b30).stop 2021-11-20T09:01:36.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2904707214 shutdown_connections 2021-11-20T09:01:36.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2904707214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78ac0f51e0 0x7f78ac0f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:36.644 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2904707214 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f4840 0x7f78ac0f4c60 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:36.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2904707214 >> 172.21.15.93:0/2904707214 conn(0x7f78ac0f0360 msgr2=0x7f78ac0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:36.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2904707214 shutdown_connections 2021-11-20T09:01:36.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.640+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2904707214 wait complete. 2021-11-20T09:01:36.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78b0aa6700 1 Processor -- start 2021-11-20T09:01:36.645 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78b0aa6700 1 -- start start 2021-11-20T09:01:36.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78b0aa6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78ac0f4840 0x7f78ac106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:36.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78b0aa6700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f51e0 0x7f78ac107310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:36.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78b0aa6700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac107910 con 0x7f78ac0f51e0 2021-11-20T09:01:36.646 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78b0aa6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac108120 con 0x7f78ac0f4840 2021-11-20T09:01:36.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78aaffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f51e0 0x7f78ac107310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:36.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78aaffd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f51e0 0x7f78ac107310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44852/0 (socket says 172.21.15.93:44852) 2021-11-20T09:01:36.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.641+0000 7f78aaffd700 1 -- 172.21.15.93:0/2990177672 learned_addr learned my addr 172.21.15.93:0/2990177672 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:36.647 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f78ab7fe700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78ac0f4840 0x7f78ac106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:36.648 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f78aaffd700 1 -- 172.21.15.93:0/2990177672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78ac0f4840 msgr2=0x7f78ac106e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:36.648 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f78aaffd700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78ac0f4840 0x7f78ac106e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:36.648 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f78aaffd700 1 -- 172.21.15.93:0/2990177672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c005040 con 0x7f78ac0f51e0 2021-11-20T09:01:36.648 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f78aaffd700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f51e0 0x7f78ac107310 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f78a0002700 tx=0x7f78a0009da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:36.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f7893fff700 1 -- 172.21.15.93:0/2990177672 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a00079c0 con 0x7f78ac0f51e0 2021-11-20T09:01:36.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f7893fff700 1 -- 172.21.15.93:0/2990177672 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78a000f040 con 0x7f78ac0f51e0 2021-11-20T09:01:36.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac1083a0 con 0x7f78ac0f51e0 2021-11-20T09:01:36.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f7893fff700 1 -- 172.21.15.93:0/2990177672 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a00148e0 con 0x7f78ac0f51e0 2021-11-20T09:01:36.649 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.642+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac108960 con 0x7f78ac0f51e0 2021-11-20T09:01:36.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.643+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac0f3ff0 con 0x7f78ac0f51e0 2021-11-20T09:01:36.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.643+0000 7f7893fff700 1 -- 172.21.15.93:0/2990177672 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f78a0007b20 con 0x7f78ac0f51e0 2021-11-20T09:01:36.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.644+0000 7f7893fff700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f789405c300 0x7f789405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:36.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.644+0000 7f7893fff700 1 -- 172.21.15.93:0/2990177672 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f78a0076580 con 0x7f78ac0f51e0 2021-11-20T09:01:36.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.644+0000 7f78ab7fe700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f789405c300 0x7f789405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:36.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.644+0000 7f78ab7fe700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f789405c300 0x7f789405e7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f78ac106b90 tx=0x7f789c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:36.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.648+0000 7f7893fff700 1 -- 172.21.15.93:0/2990177672 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78a0048c90 con 0x7f78ac0f51e0 2021-11-20T09:01:36.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.830+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78ac0fef00 con 0x7f78ac0f51e0 2021-11-20T09:01:36.831 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.831+0000 7f7893fff700 1 -- 172.21.15.93:0/2990177672 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7f78ac0fef00 con 0x7f78ac0f51e0 2021-11-20T09:01:36.832 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:36.833 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:36.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.832+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f789405c300 msgr2=0x7f789405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:36.836 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.832+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f789405c300 0x7f789405e7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f78ac106b90 tx=0x7f789c009b30).stop 2021-11-20T09:01:36.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.832+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f51e0 msgr2=0x7f78ac107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:36.837 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.832+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f51e0 0x7f78ac107310 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f78a0002700 tx=0x7f78a0009da0).stop 2021-11-20T09:01:36.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.832+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 shutdown_connections 2021-11-20T09:01:36.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.832+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f789405c300 0x7f789405e7c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:36.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.832+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78ac0f4840 0x7f78ac106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:36.840 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.833+0000 7f78b0aa6700 1 --2- 172.21.15.93:0/2990177672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78ac0f51e0 0x7f78ac107310 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:36.841 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.833+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 >> 172.21.15.93:0/2990177672 conn(0x7f78ac0f0360 msgr2=0x7f78ac0fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:36.841 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.833+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 shutdown_connections 2021-11-20T09:01:36.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:36.833+0000 7f78b0aa6700 1 -- 172.21.15.93:0/2990177672 wait complete. 2021-11-20T09:01:36.842 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:36.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:36.845 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:37.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:36 smithi102 conmon[36472]: cluster 2021-11-20T09:01:35.592665+0000 mgr.smithi093.qdikkm (mgr.14162) 331 : cluster [DBG] pgmap v311: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T09:01:37.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:36 smithi102 conmon[36472]: audit 2021-11-20T09:01:36.831926+0000 mon.smithi093 ( 2021-11-20T09:01:37.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:36 smithi102 conmon[36472]: mon.0) 928 : audit [DBG] from='client.? 172.21.15.93:0/2990177672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:37.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:36 smithi093 conmon[33522]: cluster 2021-11-20T09:01:35.592665+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T09:01:37.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:36 smithi093 conmon[33522]: 14162) 331 : cluster [DBG] pgmap v311: 65 pgs: 65 active+clean; 3.6 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-11-20T09:01:37.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:36 smithi093 conmon[33522]: audit 2021-11-20T09:01:36.831926+0000 mon.smithi093 (mon. 2021-11-20T09:01:37.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:36 smithi093 conmon[33522]: 0) 928 : audit [DBG] from='client.? 172.21.15.93:0/2990177672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:37.847 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:37.976 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 -- 172.21.15.93:0/222152848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f6810 msgr2=0x7fa9880f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:37.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 --2- 172.21.15.93:0/222152848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f6810 0x7fa9880f6c70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fa97c004660 tx=0x7fa97c009b30).stop 2021-11-20T09:01:37.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 -- 172.21.15.93:0/222152848 shutdown_connections 2021-11-20T09:01:37.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 --2- 172.21.15.93:0/222152848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f6810 0x7fa9880f6c70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:37.979 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 --2- 172.21.15.93:0/222152848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880f5570 0x7fa9880f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:37.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 -- 172.21.15.93:0/222152848 >> 172.21.15.93:0/222152848 conn(0x7fa9880f0cb0 msgr2=0x7fa9880f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:37.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 -- 172.21.15.93:0/222152848 shutdown_connections 2021-11-20T09:01:37.980 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 -- 172.21.15.93:0/222152848 wait complete. 2021-11-20T09:01:37.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 Processor -- start 2021-11-20T09:01:37.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.976+0000 7fa990546700 1 -- start start 2021-11-20T09:01:37.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa990546700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f5570 0x7fa9880604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:37.981 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa990546700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880f6810 0x7fa98810bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:37.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa990546700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98810c160 con 0x7fa9880f5570 2021-11-20T09:01:37.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa990546700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98810c2a0 con 0x7fa9880f6810 2021-11-20T09:01:37.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa98e2e2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f5570 0x7fa9880604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:37.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa98e2e2700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f5570 0x7fa9880604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44862/0 (socket says 172.21.15.93:44862) 2021-11-20T09:01:37.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa98e2e2700 1 -- 172.21.15.93:0/1623395230 learned_addr learned my addr 172.21.15.93:0/1623395230 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:37.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa98e2e2700 1 -- 172.21.15.93:0/1623395230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880f6810 msgr2=0x7fa98810bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:37.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa98e2e2700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880f6810 0x7fa98810bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:37.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa98e2e2700 1 -- 172.21.15.93:0/1623395230 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c005040 con 0x7fa9880f5570 2021-11-20T09:01:37.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa98e2e2700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f5570 0x7fa9880604f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fa97800d820 tx=0x7fa978004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:37.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa986ffd700 1 -- 172.21.15.93:0/1623395230 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa97800b510 con 0x7fa9880f5570 2021-11-20T09:01:37.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa986ffd700 1 -- 172.21.15.93:0/1623395230 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa97800f040 con 0x7fa9880f5570 2021-11-20T09:01:37.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa986ffd700 1 -- 172.21.15.93:0/1623395230 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa978013a30 con 0x7fa9880f5570 2021-11-20T09:01:37.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98810c500 con 0x7fa9880f5570 2021-11-20T09:01:37.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.977+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98810cac0 con 0x7fa9880f5570 2021-11-20T09:01:37.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.978+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa988027a00 con 0x7fa9880f5570 2021-11-20T09:01:37.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.978+0000 7fa986ffd700 1 -- 172.21.15.93:0/1623395230 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa97801a030 con 0x7fa9880f5570 2021-11-20T09:01:37.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.979+0000 7fa986ffd700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa97005c2a0 0x7fa97005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:37.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.979+0000 7fa986ffd700 1 -- 172.21.15.93:0/1623395230 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fa978076250 con 0x7fa9880f5570 2021-11-20T09:01:37.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.979+0000 7fa98dae1700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa97005c2a0 0x7fa97005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:37.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.979+0000 7fa98dae1700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa97005c2a0 0x7fa97005e760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fa97c009f50 tx=0x7fa97c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:37.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:37.983+0000 7fa986ffd700 1 -- 172.21.15.93:0/1623395230 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9780489e0 con 0x7fa9880f5570 2021-11-20T09:01:38.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.167+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9880fdd00 con 0x7fa9880f5570 2021-11-20T09:01:38.167 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.167+0000 7fa986ffd700 1 -- 172.21.15.93:0/1623395230 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7fa978018090 con 0x7fa9880f5570 2021-11-20T09:01:38.168 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:38.168 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:38.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa97005c2a0 msgr2=0x7fa97005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:38.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa97005c2a0 0x7fa97005e760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fa97c009f50 tx=0x7fa97c009b30).stop 2021-11-20T09:01:38.173 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f5570 msgr2=0x7fa9880604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:38.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f5570 0x7fa9880604f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fa97800d820 tx=0x7fa978004e80).stop 2021-11-20T09:01:38.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 shutdown_connections 2021-11-20T09:01:38.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fa97005c2a0 0x7fa97005e760 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:38.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9880f5570 0x7fa9880604f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:38.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 --2- 172.21.15.93:0/1623395230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9880f6810 0x7fa98810bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:38.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 >> 172.21.15.93:0/1623395230 conn(0x7fa9880f0cb0 msgr2=0x7fa9880fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:38.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 shutdown_connections 2021-11-20T09:01:38.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:38.169+0000 7fa990546700 1 -- 172.21.15.93:0/1623395230 wait complete. 2021-11-20T09:01:38.175 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:38.179 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:38.180 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:39.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:38 smithi102 conmon[36472]: cluster 2021-11-20T09:01:37.593224+0000 mgr.smithi093.qdikkm (mgr.14162) 332 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:39.170 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:38 smithi102 conmon[36472]: audit 2021-11-20T09:01:38.168334+0000 mon.smithi093 (mon.0) 929 : audit [DBG] from='client.? 172.21.15.93:0/1623395230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:39.182 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:39.210 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:38 smithi093 conmon[33522]: cluster 2021-11-20T09:01:37.593224+0000 mgr.smithi093.qdikkm (mgr.14162) 332 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T09:01:39.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:38 smithi093 conmon[33522]: audit 2021-11-20T09:01:38.168334+0000 mon.smithi093 (mon.0) 929 : audit [DBG] from='client.? 172.21.15.93:0/1623395230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:39.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 -- 172.21.15.93:0/3216674359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 msgr2=0x7f05a40f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:39.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 --2- 172.21.15.93:0/3216674359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a40f6ac0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f05940099e0 tx=0x7f0594004d10).stop 2021-11-20T09:01:39.310 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 -- 172.21.15.93:0/3216674359 shutdown_connections 2021-11-20T09:01:39.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 --2- 172.21.15.93:0/3216674359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05a40f7940 0x7f05a40f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:39.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 --2- 172.21.15.93:0/3216674359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a40f6ac0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:39.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 -- 172.21.15.93:0/3216674359 >> 172.21.15.93:0/3216674359 conn(0x7f05a40f1da0 msgr2=0x7f05a40f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:39.311 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 -- 172.21.15.93:0/3216674359 shutdown_connections 2021-11-20T09:01:39.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 -- 172.21.15.93:0/3216674359 wait complete. 2021-11-20T09:01:39.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 Processor -- start 2021-11-20T09:01:39.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.309+0000 7f05a90ca700 1 -- start start 2021-11-20T09:01:39.312 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a90ca700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a4106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:39.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a90ca700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05a40f7940 0x7f05a41074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:39.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a90ca700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4107a50 con 0x7f05a40f66a0 2021-11-20T09:01:39.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a90ca700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4107b90 con 0x7f05a40f7940 2021-11-20T09:01:39.314 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a2d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a4106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:39.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a2d9d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a4106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44872/0 (socket says 172.21.15.93:44872) 2021-11-20T09:01:39.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a2d9d700 1 -- 172.21.15.93:0/1970286910 learned_addr learned my addr 172.21.15.93:0/1970286910 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:39.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a2d9d700 1 -- 172.21.15.93:0/1970286910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05a40f7940 msgr2=0x7f05a41074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:39.315 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a2d9d700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05a40f7940 0x7f05a41074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:39.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a2d9d700 1 -- 172.21.15.93:0/1970286910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594009640 con 0x7f05a40f66a0 2021-11-20T09:01:39.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a2d9d700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a4106fb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f0594007830 tx=0x7f0594004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:39.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f059b7fe700 1 -- 172.21.15.93:0/1970286910 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f059400e030 con 0x7f05a40f66a0 2021-11-20T09:01:39.316 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a410c5d0 con 0x7f05a40f66a0 2021-11-20T09:01:39.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f059b7fe700 1 -- 172.21.15.93:0/1970286910 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f059401d8d0 con 0x7f05a40f66a0 2021-11-20T09:01:39.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f059b7fe700 1 -- 172.21.15.93:0/1970286910 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0594018070 con 0x7f05a40f66a0 2021-11-20T09:01:39.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.310+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a410cb10 con 0x7f05a40f66a0 2021-11-20T09:01:39.317 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.311+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a4027a00 con 0x7f05a40f66a0 2021-11-20T09:01:39.318 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.311+0000 7f059b7fe700 1 -- 172.21.15.93:0/1970286910 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f059401da40 con 0x7f05a40f66a0 2021-11-20T09:01:39.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.312+0000 7f059b7fe700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f059005c2a0 0x7f059005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:39.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.312+0000 7f059b7fe700 1 -- 172.21.15.93:0/1970286910 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f05940783c0 con 0x7f05a40f66a0 2021-11-20T09:01:39.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.312+0000 7f05a259c700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f059005c2a0 0x7f059005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:39.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.312+0000 7f05a259c700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f059005c2a0 0x7f059005e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f058c0047d0 tx=0x7f058c007a40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:39.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.316+0000 7f059b7fe700 1 -- 172.21.15.93:0/1970286910 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f059404aad0 con 0x7f05a40f66a0 2021-11-20T09:01:39.499 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.500+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05a40fee30 con 0x7f05a40f66a0 2021-11-20T09:01:39.500 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.501+0000 7f059b7fe700 1 -- 172.21.15.93:0/1970286910 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7f059404e0f0 con 0x7f05a40f66a0 2021-11-20T09:01:39.500 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:39.501 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:39.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.502+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f059005c2a0 msgr2=0x7f059005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:39.502 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.502+0000 7f05a90ca700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f059005c2a0 0x7f059005e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f058c0047d0 tx=0x7f058c007a40).stop 2021-11-20T09:01:39.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.502+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 msgr2=0x7f05a4106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:39.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.502+0000 7f05a90ca700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a4106fb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f0594007830 tx=0x7f0594004d10).stop 2021-11-20T09:01:39.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.502+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 shutdown_connections 2021-11-20T09:01:39.503 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.503+0000 7f05a90ca700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f059005c2a0 0x7f059005e760 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:39.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.503+0000 7f05a90ca700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05a40f66a0 0x7f05a4106fb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:39.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.503+0000 7f05a90ca700 1 --2- 172.21.15.93:0/1970286910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05a40f7940 0x7f05a41074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:39.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.503+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 >> 172.21.15.93:0/1970286910 conn(0x7f05a40f1da0 msgr2=0x7f05a40fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:39.504 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.503+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 shutdown_connections 2021-11-20T09:01:39.505 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:39.503+0000 7f05a90ca700 1 -- 172.21.15.93:0/1970286910 wait complete. 2021-11-20T09:01:39.505 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:39.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:39.520 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:40.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:39 smithi102 conmon[36472]: audit 2021-11-20T09:01:39.502019+0000 mon.smithi093 (mon.0) 930 : audit [DBG] from='client.? 172.21.15.93:0/1970286910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:40.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:39 smithi093 conmon[33522]: audit 2021-11-20T09:01:39.502019+0000 mon.smithi093 (mon. 2021-11-20T09:01:40.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:39 smithi093 conmon[33522]: 0) 930 : audit [DBG] from='client.? 172.21.15.93:0/1970286910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:40.522 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:40.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 -- 172.21.15.93:0/610206860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f66a0 msgr2=0x7f61ac0f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:40.650 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 --2- 172.21.15.93:0/610206860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f66a0 0x7f61ac0f6ac0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f61a8004660 tx=0x7f61a8009b30).stop 2021-11-20T09:01:40.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 -- 172.21.15.93:0/610206860 shutdown_connections 2021-11-20T09:01:40.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 --2- 172.21.15.93:0/610206860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac0f7940 0x7f61ac0f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:40.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 --2- 172.21.15.93:0/610206860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f66a0 0x7f61ac0f6ac0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:40.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 -- 172.21.15.93:0/610206860 >> 172.21.15.93:0/610206860 conn(0x7f61ac0f1da0 msgr2=0x7f61ac0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:40.651 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 -- 172.21.15.93:0/610206860 shutdown_connections 2021-11-20T09:01:40.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 -- 172.21.15.93:0/610206860 wait complete. 2021-11-20T09:01:40.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 Processor -- start 2021-11-20T09:01:40.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.649+0000 7f61b46ca700 1 -- start start 2021-11-20T09:01:40.652 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b46ca700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac0f66a0 0x7f61ac106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:40.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b46ca700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f7940 0x7f61ac107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:40.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b46ca700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac107940 con 0x7f61ac0f7940 2021-11-20T09:01:40.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b46ca700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac107a80 con 0x7f61ac0f66a0 2021-11-20T09:01:40.653 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b1c65700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f7940 0x7f61ac107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:40.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b1c65700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f7940 0x7f61ac107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44884/0 (socket says 172.21.15.93:44884) 2021-11-20T09:01:40.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b1c65700 1 -- 172.21.15.93:0/1555813103 learned_addr learned my addr 172.21.15.93:0/1555813103 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:40.654 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b2466700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac0f66a0 0x7f61ac106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:40.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b1c65700 1 -- 172.21.15.93:0/1555813103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac0f66a0 msgr2=0x7f61ac106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:40.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b1c65700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac0f66a0 0x7f61ac106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:40.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b1c65700 1 -- 172.21.15.93:0/1555813103 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8005040 con 0x7f61ac0f7940 2021-11-20T09:01:40.655 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b2466700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac0f66a0 0x7f61ac106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T09:01:40.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b1c65700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f7940 0x7f61ac107390 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f619c0026d0 tx=0x7f619c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:40.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61a2ffd700 1 -- 172.21.15.93:0/1555813103 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f619c009320 con 0x7f61ac0f7940 2021-11-20T09:01:40.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61a2ffd700 1 -- 172.21.15.93:0/1555813103 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f619c00f040 con 0x7f61ac0f7940 2021-11-20T09:01:40.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ac10c520 con 0x7f61ac0f7940 2021-11-20T09:01:40.656 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61a2ffd700 1 -- 172.21.15.93:0/1555813103 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f619c0147e0 con 0x7f61ac0f7940 2021-11-20T09:01:40.657 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.650+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac10cae0 con 0x7f61ac0f7940 2021-11-20T09:01:40.658 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.651+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac027a00 con 0x7f61ac0f7940 2021-11-20T09:01:40.658 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.652+0000 7f61a2ffd700 1 -- 172.21.15.93:0/1555813103 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f619c009480 con 0x7f61ac0f7940 2021-11-20T09:01:40.659 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.652+0000 7f61a2ffd700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f619805c2a0 0x7f619805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:40.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.652+0000 7f61a2ffd700 1 -- 172.21.15.93:0/1555813103 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f619c076ba0 con 0x7f61ac0f7940 2021-11-20T09:01:40.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.652+0000 7f61b2466700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f619805c2a0 0x7f619805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:40.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.652+0000 7f61b2466700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f619805c2a0 0x7f619805e760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f61a8009d60 tx=0x7f61a8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:40.660 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.656+0000 7f61a2ffd700 1 -- 172.21.15.93:0/1555813103 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f619c0492b0 con 0x7f61ac0f7940 2021-11-20T09:01:40.838 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.839+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61ac0fee30 con 0x7f61ac0f7940 2021-11-20T09:01:40.839 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.840+0000 7f61a2ffd700 1 -- 172.21.15.93:0/1555813103 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7f619c04c8d0 con 0x7f61ac0f7940 2021-11-20T09:01:40.839 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:40.839 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:40.841 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f619805c2a0 msgr2=0x7f619805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:40.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f619805c2a0 0x7f619805e760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f61a8009d60 tx=0x7f61a8009b30).stop 2021-11-20T09:01:40.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f7940 msgr2=0x7f61ac107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:40.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f7940 0x7f61ac107390 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f619c0026d0 tx=0x7f619c009de0).stop 2021-11-20T09:01:40.842 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 shutdown_connections 2021-11-20T09:01:40.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f619805c2a0 0x7f619805e760 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:40.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac0f66a0 0x7f61ac106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:40.843 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 --2- 172.21.15.93:0/1555813103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61ac0f7940 0x7f61ac107390 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:40.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 >> 172.21.15.93:0/1555813103 conn(0x7f61ac0f1da0 msgr2=0x7f61ac0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:40.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 shutdown_connections 2021-11-20T09:01:40.844 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:40.841+0000 7f61b46ca700 1 -- 172.21.15.93:0/1555813103 wait complete. 2021-11-20T09:01:40.844 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:40.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:40.853 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:41.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:40 smithi102 conmon[36472]: cluster 2021-11-20T09:01:39.593822+0000 mgr.smithi093.qdikkm (mgr.14162) 333 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 201 B/s wr, 1 op/s 2021-11-20T09:01:41.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:40 smithi102 conmon[36472]: audit 2021-11-20T 2021-11-20T09:01:41.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:40 smithi102 conmon[36472]: 09:01:40.840637+0000 mon.smithi093 (mon.0) 931 : audit [DBG] from='client.? 172.21.15.93:0/1555813103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:41.189 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:40 smithi093 conmon[33522]: cluster 2021-11-20T09:01:39.593822+0000 mgr.smithi093.qdikkm 2021-11-20T09:01:41.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:40 smithi093 conmon[33522]: (mgr.14162) 333 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 201 B/s wr, 1 op/s 2021-11-20T09:01:41.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:40 smithi093 conmon[33522]: audit 2021-11-20T09:01:40.840637+0000 2021-11-20T09:01:41.190 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:40 smithi093 conmon[33522]: mon.smithi093 (mon.0) 931 : audit [DBG] from='client.? 172.21.15.93:0/1555813103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:41.854 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:41.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/2336395243 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f4960 msgr2=0x7fd59c0f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:41.982 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/2336395243 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f4960 0x7fd59c0f4d80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fd58c004660 tx=0x7fd58c009b30).stop 2021-11-20T09:01:41.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/2336395243 shutdown_connections 2021-11-20T09:01:41.983 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/2336395243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd59c0f5300 0x7fd59c0f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:41.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/2336395243 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f4960 0x7fd59c0f4d80 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:41.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/2336395243 >> 172.21.15.93:0/2336395243 conn(0x7fd59c0f0440 msgr2=0x7fd59c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:41.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/2336395243 shutdown_connections 2021-11-20T09:01:41.984 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/2336395243 wait complete. 2021-11-20T09:01:41.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.980+0000 7fd5a3f4d700 1 Processor -- start 2021-11-20T09:01:41.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a3f4d700 1 -- start start 2021-11-20T09:01:41.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a3f4d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd59c0f4960 0x7fd59c0f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:41.985 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a3f4d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f5300 0x7fd59c0f5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:41.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a3f4d700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0f78e0 con 0x7fd59c0f5300 2021-11-20T09:01:41.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a3f4d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0f5e80 con 0x7fd59c0f4960 2021-11-20T09:01:41.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a14e8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f5300 0x7fd59c0f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:41.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a14e8700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f5300 0x7fd59c0f5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44896/0 (socket says 172.21.15.93:44896) 2021-11-20T09:01:41.986 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a14e8700 1 -- 172.21.15.93:0/59867806 learned_addr learned my addr 172.21.15.93:0/59867806 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:41.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a14e8700 1 -- 172.21.15.93:0/59867806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd59c0f4960 msgr2=0x7fd59c0f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:41.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a14e8700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd59c0f4960 0x7fd59c0f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:41.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a14e8700 1 -- 172.21.15.93:0/59867806 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c005040 con 0x7fd59c0f5300 2021-11-20T09:01:41.987 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a14e8700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f5300 0x7fd59c0f5990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fd598002700 tx=0x7fd598009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:41.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5927fc700 1 -- 172.21.15.93:0/59867806 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd598009320 con 0x7fd59c0f5300 2021-11-20T09:01:41.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c0f6100 con 0x7fd59c0f5300 2021-11-20T09:01:41.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5927fc700 1 -- 172.21.15.93:0/59867806 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd59800f040 con 0x7fd59c0f5300 2021-11-20T09:01:41.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5927fc700 1 -- 172.21.15.93:0/59867806 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5980147c0 con 0x7fd59c0f5300 2021-11-20T09:01:41.988 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.981+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c0f66c0 con 0x7fd59c0f5300 2021-11-20T09:01:41.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.982+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c027a00 con 0x7fd59c0f5300 2021-11-20T09:01:41.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.983+0000 7fd5927fc700 1 -- 172.21.15.93:0/59867806 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd5980077b0 con 0x7fd59c0f5300 2021-11-20T09:01:41.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.983+0000 7fd5927fc700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd58805c2a0 0x7fd58805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:41.989 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.983+0000 7fd5927fc700 1 -- 172.21.15.93:0/59867806 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fd598076ad0 con 0x7fd59c0f5300 2021-11-20T09:01:41.990 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.983+0000 7fd5a1ce9700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd58805c2a0 0x7fd58805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:41.990 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.983+0000 7fd5a1ce9700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd58805c2a0 0x7fd58805e760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fd58c009f50 tx=0x7fd58c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:41.991 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:41.987+0000 7fd5927fc700 1 -- 172.21.15.93:0/59867806 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5980491e0 con 0x7fd59c0f5300 2021-11-20T09:01:42.170 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.171+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd59c0fee70 con 0x7fd59c0f5300 2021-11-20T09:01:42.171 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.172+0000 7fd5927fc700 1 -- 172.21.15.93:0/59867806 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7fd59804c800 con 0x7fd59c0f5300 2021-11-20T09:01:42.171 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:42.172 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:42.174 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd58805c2a0 msgr2=0x7fd58805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:42.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd58805c2a0 0x7fd58805e760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fd58c009f50 tx=0x7fd58c009b30).stop 2021-11-20T09:01:42.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f5300 msgr2=0x7fd59c0f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:42.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f5300 0x7fd59c0f5990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fd598002700 tx=0x7fd598009d60).stop 2021-11-20T09:01:42.175 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 shutdown_connections 2021-11-20T09:01:42.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7fd58805c2a0 0x7fd58805e760 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:42.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd59c0f4960 0x7fd59c0f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T09:01:42.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 --2- 172.21.15.93:0/59867806 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd59c0f5300 0x7fd59c0f5990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:42.176 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.173+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 >> 172.21.15.93:0/59867806 conn(0x7fd59c0f0440 msgr2=0x7fd59c0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:42.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.174+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 shutdown_connections 2021-11-20T09:01:42.177 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:42.174+0000 7fd5a3f4d700 1 -- 172.21.15.93:0/59867806 wait complete. 2021-11-20T09:01:42.177 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:42.185 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:42.186 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:43.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:42 smithi102 conmon[36472]: cluster 2021-11-20T09:01:41.594304+0000 mgr.smithi093.qdikkm (mgr.14162) 334 : cluster [DBG] pgmap v314: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:01:43.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:42 smithi102 conmon[36472]: audit 2021-11-20T09:01:42.172987+0000 2021-11-20T09:01:43.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:42 smithi102 conmon[36472]: mon.smithi093 (mon.0) 932 : audit [DBG] from='client.? 172.21.15.93:0/59867806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:43.189 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:43.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:42 smithi093 conmon[33522]: cluster 2021-11-20T09:01:41.594304+0000 mgr.smithi093.qdikkm (mgr.14162) 334 : cluster 2021-11-20T09:01:43.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:42 smithi093 conmon[33522]: [DBG] pgmap v314: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T09:01:43.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:42 smithi093 conmon[33522]: audit 2021-11-20T09:01:42.172987+0000 mon.smithi093 (mon.0) 932 : audit [DBG] from='client.? 172.21.15.93:0/59867806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:43.319 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 -- 172.21.15.93:0/2821189772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 msgr2=0x7f846c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 --2- 172.21.15.93:0/2821189772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c0f59a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f845c004660 tx=0x7f845c009b30).stop 2021-11-20T09:01:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 -- 172.21.15.93:0/2821189772 shutdown_connections 2021-11-20T09:01:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 --2- 172.21.15.93:0/2821189772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f846c0f6820 0x7f846c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 --2- 172.21.15.93:0/2821189772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c0f59a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:43.320 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 -- 172.21.15.93:0/2821189772 >> 172.21.15.93:0/2821189772 conn(0x7f846c0f0ce0 msgr2=0x7f846c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:43.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 -- 172.21.15.93:0/2821189772 shutdown_connections 2021-11-20T09:01:43.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 -- 172.21.15.93:0/2821189772 wait complete. 2021-11-20T09:01:43.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 Processor -- start 2021-11-20T09:01:43.321 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 -- start start 2021-11-20T09:01:43.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.316+0000 7f84720cd700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:43.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f84720cd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f846c0f6820 0x7f846c10bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:43.322 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f84720cd700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c10c110 con 0x7f846c0f5580 2021-11-20T09:01:43.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f84720cd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c10c250 con 0x7f846c0f6820 2021-11-20T09:01:43.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:43.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846b7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c060510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44904/0 (socket says 172.21.15.93:44904) 2021-11-20T09:01:43.323 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846b7fe700 1 -- 172.21.15.93:0/716687853 learned_addr learned my addr 172.21.15.93:0/716687853 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:43.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846affd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f846c0f6820 0x7f846c10bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:43.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846b7fe700 1 -- 172.21.15.93:0/716687853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f846c0f6820 msgr2=0x7f846c10bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:43.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846b7fe700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f846c0f6820 0x7f846c10bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:43.324 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846b7fe700 1 -- 172.21.15.93:0/716687853 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c005040 con 0x7f846c0f5580 2021-11-20T09:01:43.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f846b7fe700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c060510 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f845c009f50 tx=0x7f845c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:43.325 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f8453fff700 1 -- 172.21.15.93:0/716687853 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845c018070 con 0x7f846c0f5580 2021-11-20T09:01:43.326 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c10c450 con 0x7f846c0f5580 2021-11-20T09:01:43.327 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f8453fff700 1 -- 172.21.15.93:0/716687853 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f845c013410 con 0x7f846c0f5580 2021-11-20T09:01:43.328 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f8453fff700 1 -- 172.21.15.93:0/716687853 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845c01de70 con 0x7f846c0f5580 2021-11-20T09:01:43.329 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.317+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c10c9b0 con 0x7f846c0f5580 2021-11-20T09:01:43.330 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.318+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c027a00 con 0x7f846c0f5580 2021-11-20T09:01:43.331 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.318+0000 7f8453fff700 1 -- 172.21.15.93:0/716687853 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f845c01d420 con 0x7f846c0f5580 2021-11-20T09:01:43.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.319+0000 7f8453fff700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f845405c2a0 0x7f845405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:43.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.319+0000 7f8453fff700 1 -- 172.21.15.93:0/716687853 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f845c0780c0 con 0x7f846c0f5580 2021-11-20T09:01:43.332 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.319+0000 7f846affd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f845405c2a0 0x7f845405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:43.333 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.319+0000 7f846affd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f845405c2a0 0x7f845405e760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f8460009740 tx=0x7f8460009c60).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:43.334 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.323+0000 7f8453fff700 1 -- 172.21.15.93:0/716687853 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f845c04a7d0 con 0x7f846c0f5580 2021-11-20T09:01:43.507 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.506+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f846c0fdd10 con 0x7f846c0f5580 2021-11-20T09:01:43.508 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.506+0000 7f8453fff700 1 -- 172.21.15.93:0/716687853 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7f845c04ddf0 con 0x7f846c0f5580 2021-11-20T09:01:43.516 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:43.516 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:43.529 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f845405c2a0 msgr2=0x7f845405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:43.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f845405c2a0 0x7f845405e760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f8460009740 tx=0x7f8460009c60).stop 2021-11-20T09:01:43.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 msgr2=0x7f846c060510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:43.531 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c060510 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f845c009f50 tx=0x7f845c009b30).stop 2021-11-20T09:01:43.532 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 shutdown_connections 2021-11-20T09:01:43.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f845405c2a0 0x7f845405e760 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:43.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f846c0f5580 0x7f846c060510 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:43.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 --2- 172.21.15.93:0/716687853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f846c0f6820 0x7f846c10bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:43.533 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 >> 172.21.15.93:0/716687853 conn(0x7f846c0f0ce0 msgr2=0x7f846c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:43.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 shutdown_connections 2021-11-20T09:01:43.534 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:43.508+0000 7f84720cd700 1 -- 172.21.15.93:0/716687853 wait complete. 2021-11-20T09:01:43.534 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:43.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:43.535 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:44.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:43 smithi102 conmon[36472]: audit 2021-11-20T09:01:43.507100+0000 mon.smithi093 (mon.0) 933 2021-11-20T09:01:44.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:43 smithi102 conmon[36472]: : audit [DBG] from='client.? 172.21.15.93:0/716687853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:44.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:43 smithi093 conmon[33522]: audit 2021-11-20T09:01:43.507100+0000 mon.smithi093 (mon.0) 933 : audit [DBG] from='client.? 172.21.15.93:0/716687853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:44.539 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:44.672 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 -- 172.21.15.93:0/2107562169 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 msgr2=0x7f44900f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:44.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 --2- 172.21.15.93:0/2107562169 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f44900f59c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f44800099e0 tx=0x7f4480004d10).stop 2021-11-20T09:01:44.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 -- 172.21.15.93:0/2107562169 shutdown_connections 2021-11-20T09:01:44.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 --2- 172.21.15.93:0/2107562169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44900f6840 0x7f44900f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:44.673 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 --2- 172.21.15.93:0/2107562169 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f44900f59c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:44.674 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 -- 172.21.15.93:0/2107562169 >> 172.21.15.93:0/2107562169 conn(0x7f44900f0ce0 msgr2=0x7f44900f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:44.674 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 -- 172.21.15.93:0/2107562169 shutdown_connections 2021-11-20T09:01:44.674 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 -- 172.21.15.93:0/2107562169 wait complete. 2021-11-20T09:01:44.674 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 Processor -- start 2021-11-20T09:01:44.675 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.672+0000 7f4495c96700 1 -- start start 2021-11-20T09:01:44.675 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4495c96700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f4490105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:44.675 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4495c96700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44900f6840 0x7f44901063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:44.675 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4495c96700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4490106950 con 0x7f44900f55a0 2021-11-20T09:01:44.676 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4495c96700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4490106a90 con 0x7f44900f6840 2021-11-20T09:01:44.676 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f4490105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:44.676 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448f7fe700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f4490105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44914/0 (socket says 172.21.15.93:44914) 2021-11-20T09:01:44.676 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448f7fe700 1 -- 172.21.15.93:0/375452215 learned_addr learned my addr 172.21.15.93:0/375452215 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:44.676 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448effd700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44900f6840 0x7f44901063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:44.677 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448f7fe700 1 -- 172.21.15.93:0/375452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44900f6840 msgr2=0x7f44901063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:44.677 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448f7fe700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44900f6840 0x7f44901063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:44.677 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448f7fe700 1 -- 172.21.15.93:0/375452215 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4480009640 con 0x7f44900f55a0 2021-11-20T09:01:44.677 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f448f7fe700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f4490105eb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f44800078b0 tx=0x7f4480004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:44.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4477fff700 1 -- 172.21.15.93:0/375452215 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f448000e030 con 0x7f44900f55a0 2021-11-20T09:01:44.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4477fff700 1 -- 172.21.15.93:0/375452215 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4480007d50 con 0x7f44900f55a0 2021-11-20T09:01:44.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449010b4d0 con 0x7f44900f55a0 2021-11-20T09:01:44.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4477fff700 1 -- 172.21.15.93:0/375452215 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4480018070 con 0x7f44900f55a0 2021-11-20T09:01:44.678 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.673+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449010ba10 con 0x7f44900f55a0 2021-11-20T09:01:44.679 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.674+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4490027a00 con 0x7f44900f55a0 2021-11-20T09:01:44.680 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.675+0000 7f4477fff700 1 -- 172.21.15.93:0/375452215 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f44800137e0 con 0x7f44900f55a0 2021-11-20T09:01:44.680 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.675+0000 7f4477fff700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f447805c2a0 0x7f447805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:44.680 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.675+0000 7f4477fff700 1 -- 172.21.15.93:0/375452215 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f4480078210 con 0x7f44900f55a0 2021-11-20T09:01:44.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.675+0000 7f448effd700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f447805c2a0 0x7f447805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:44.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.675+0000 7f448effd700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f447805c2a0 0x7f447805e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4484004c40 tx=0x7f448400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:44.681 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.679+0000 7f4477fff700 1 -- 172.21.15.93:0/375452215 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f448004a9d0 con 0x7f44900f55a0 2021-11-20T09:01:44.861 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.861+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44900fdd30 con 0x7f44900f55a0 2021-11-20T09:01:44.862 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.861+0000 7f4477fff700 1 -- 172.21.15.93:0/375452215 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7f448004dff0 con 0x7f44900f55a0 2021-11-20T09:01:44.863 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:44.875 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:44.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.862+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f447805c2a0 msgr2=0x7f447805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:44.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.862+0000 7f4495c96700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f447805c2a0 0x7f447805e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4484004c40 tx=0x7f448400b040).stop 2021-11-20T09:01:44.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.862+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 msgr2=0x7f4490105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:44.877 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.862+0000 7f4495c96700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f4490105eb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f44800078b0 tx=0x7f4480004d10).stop 2021-11-20T09:01:44.878 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.863+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 shutdown_connections 2021-11-20T09:01:44.878 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.863+0000 7f4495c96700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7f447805c2a0 0x7f447805e760 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:44.879 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.863+0000 7f4495c96700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44900f55a0 0x7f4490105eb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:44.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.863+0000 7f4495c96700 1 --2- 172.21.15.93:0/375452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44900f6840 0x7f44901063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:44.880 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.863+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 >> 172.21.15.93:0/375452215 conn(0x7f44900f0ce0 msgr2=0x7f44900fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:44.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.863+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 shutdown_connections 2021-11-20T09:01:44.881 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:44.863+0000 7f4495c96700 1 -- 172.21.15.93:0/375452215 wait complete. 2021-11-20T09:01:44.881 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:44.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:44.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:45.167 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:44 smithi102 conmon[36472]: cluster 2021-11-20T09:01:43.594908+0000 mgr.smithi093.qdikkm (mgr. 2021-11-20T09:01:45.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:44 smithi102 conmon[36472]: 14162) 335 : cluster [DBG] pgmap v315: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:01:45.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:44 smithi102 conmon[36472]: audit 2021-11-20T09:01:44.862071+0000 mon.smithi093 (mon.0) 934 : audit [DBG] from='client.? 172.21.15.93:0/375452215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:45.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:44 smithi093 conmon[33522]: cluster 2021-11-20T09:01:43.594908+0000 mgr.smithi093.qdikkm (mgr.14162) 335 2021-11-20T09:01:45.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:44 smithi093 conmon[33522]: : cluster [DBG] pgmap v315: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:01:45.212 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:44 smithi093 conmon[33522]: audit 2021-11-20T09:01:44.862071+0000 mon.smithi093 (mon.0) 934 : audit [DBG] from='client.? 172.21.15.93:0/375452215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:45.886 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T09:01:46.023 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.022+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2391245802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f7940 msgr2=0x7ff3e40f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:46.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.022+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2391245802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f7940 0x7ff3e40f7da0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7ff3d0004660 tx=0x7ff3d0009b30).stop 2021-11-20T09:01:46.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.022+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2391245802 shutdown_connections 2021-11-20T09:01:46.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.022+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2391245802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f7940 0x7ff3e40f7da0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:46.024 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.022+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2391245802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e40f66a0 0x7ff3e40f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:46.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.022+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2391245802 >> 172.21.15.93:0/2391245802 conn(0x7ff3e40f1e00 msgr2=0x7ff3e40f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:46.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2391245802 shutdown_connections 2021-11-20T09:01:46.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2391245802 wait complete. 2021-11-20T09:01:46.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 Processor -- start 2021-11-20T09:01:46.025 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 -- start start 2021-11-20T09:01:46.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f66a0 0x7ff3e4106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:46.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e40f7940 0x7ff3e4107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:46.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e4107900 con 0x7ff3e40f66a0 2021-11-20T09:01:46.026 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.023+0000 7ff3eb0c1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e4107a40 con 0x7ff3e40f7940 2021-11-20T09:01:46.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e8e5d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f66a0 0x7ff3e4106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:46.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e8e5d700 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f66a0 0x7ff3e4106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:44924/0 (socket says 172.21.15.93:44924) 2021-11-20T09:01:46.027 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e8e5d700 1 -- 172.21.15.93:0/2483607766 learned_addr learned my addr 172.21.15.93:0/2483607766 (peer_addr_for_me v2:172.21.15.93:0/0) 2021-11-20T09:01:46.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e3fff700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e40f7940 0x7ff3e4107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:46.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e8e5d700 1 -- 172.21.15.93:0/2483607766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e40f7940 msgr2=0x7ff3e4107350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:46.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e8e5d700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e40f7940 0x7ff3e4107350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:46.028 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e8e5d700 1 -- 172.21.15.93:0/2483607766 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0005040 con 0x7ff3e40f66a0 2021-11-20T09:01:46.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e8e5d700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f66a0 0x7ff3e4106e60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7ff3d8002700 tx=0x7ff3d8009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:46.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e17fa700 1 -- 172.21.15.93:0/2483607766 <== mon.0 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d80079c0 con 0x7ff3e40f66a0 2021-11-20T09:01:46.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e410c4e0 con 0x7ff3e40f66a0 2021-11-20T09:01:46.029 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e17fa700 1 -- 172.21.15.93:0/2483607766 <== mon.0 v2:172.21.15.93:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3d800f040 con 0x7ff3e40f66a0 2021-11-20T09:01:46.030 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3e17fa700 1 -- 172.21.15.93:0/2483607766 <== mon.0 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d8014860 con 0x7ff3e40f66a0 2021-11-20T09:01:46.030 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.024+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e410caa0 con 0x7ff3e40f66a0 2021-11-20T09:01:46.031 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.025+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e4027a00 con 0x7ff3e40f66a0 2021-11-20T09:01:46.032 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.026+0000 7ff3e17fa700 1 -- 172.21.15.93:0/2483607766 <== mon.0 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff3d801b060 con 0x7ff3e40f66a0 2021-11-20T09:01:46.032 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.026+0000 7ff3e17fa700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3d405c2a0 0x7ff3d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T09:01:46.032 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.026+0000 7ff3e17fa700 1 -- 172.21.15.93:0/2483607766 <== mon.0 v2:172.21.15.93:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6844+0+0 (secure 0 0 0) 0x7ff3d8076340 con 0x7ff3e40f66a0 2021-11-20T09:01:46.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.026+0000 7ff3e3fff700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3d405c2a0 0x7ff3d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T09:01:46.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.026+0000 7ff3e3fff700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3d405c2a0 0x7ff3d405e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7ff3d0009f50 tx=0x7ff3d0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T09:01:46.033 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.030+0000 7ff3e17fa700 1 -- 172.21.15.93:0/2483607766 <== mon.0 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3d8048a50 con 0x7ff3e40f66a0 2021-11-20T09:01:46.214 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.215+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e40fee30 con 0x7ff3e40f66a0 2021-11-20T09:01:46.214 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.215+0000 7ff3e17fa700 1 -- 172.21.15.93:0/2483607766 <== mon.0 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2194 (secure 0 0 0) 0x7ff3d804c070 con 0x7ff3e40f66a0 2021-11-20T09:01:46.215 INFO:teuthology.orchestra.run.smithi093.stdout: 2021-11-20T09:01:46.216 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24649,"name":"cephfs.smithi102.hjodmi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.102:6827/3890204616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":3890204616},{"type":"v1","addr":"172.21.15.102:6827","nonce":3890204616}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T08:56:42.318132+0000","modified":"2021-11-20T09:01:32.855707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24623":{"gid":24623,"name":"cephfs.smithi102.lhrlmh","rank":0,"incarnation":98,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.102:6825/3315610363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":3315610363},{"type":"v1","addr":"172.21.15.102:6825","nonce":3315610363}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T09:01:46.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3d405c2a0 msgr2=0x7ff3d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:46.218 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3d405c2a0 0x7ff3d405e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7ff3d0009f50 tx=0x7ff3d0009b30).stop 2021-11-20T09:01:46.219 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f66a0 msgr2=0x7ff3e4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T09:01:46.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f66a0 0x7ff3e4106e60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7ff3d8002700 tx=0x7ff3d8009d80).stop 2021-11-20T09:01:46.220 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 shutdown_connections 2021-11-20T09:01:46.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:6800/3953495879,v1:172.21.15.93:6801/3953495879] conn(0x7ff3d405c2a0 0x7ff3d405e760 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:46.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3e40f66a0 0x7ff3e4106e60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:46.221 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 --2- 172.21.15.93:0/2483607766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3e40f7940 0x7ff3e4107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T09:01:46.222 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 >> 172.21.15.93:0/2483607766 conn(0x7ff3e40f1e00 msgr2=0x7ff3e40fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T09:01:46.223 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 shutdown_connections 2021-11-20T09:01:46.224 INFO:teuthology.orchestra.run.smithi093.stderr:2021-11-20T09:01:46.217+0000 7ff3eb0c1700 1 -- 172.21.15.93:0/2483607766 wait complete. 2021-11-20T09:01:46.224 INFO:teuthology.orchestra.run.smithi093.stderr:dumped fsmap epoch 102 2021-11-20T09:01:46.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T08:56:42.318132+0000', 'modified': '2021-11-20T09:01:32.855707+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24623': {'gid': 24623, 'name': 'cephfs.smithi102.lhrlmh', 'rank': 0, 'incarnation': 98, 'state': 'up:clientreplay', 'state_seq': 7, 'addr': '172.21.15.102:6825/3315610363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6824', 'nonce': 3315610363}, {'type': 'v1', 'addr': '172.21.15.102:6825', 'nonce': 3315610363}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T09:01:46.228 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24623:up:clientreplay 2021-11-20T09:01:47.168 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:46 smithi102 conmon[36472]: cluster 2021-11-20T09:01:45.595463+0000 mgr.smithi093.qdikkm (mgr.14162) 336 : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:01:47.169 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:46 smithi102 conmon[36472]: audit 2021-11-20T09:01:46.216206+0000 mon.smithi093 (mon.0) 935 : audit [DBG] from='client.? 172.21.15.93:0/2483607766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:47.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:46 smithi093 conmon[33522]: cluster 2021-11-20T09:01:45.595463+0000 mgr.smithi093.qdikkm (mgr.14162) 336 2021-11-20T09:01:47.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:46 smithi093 conmon[33522]: : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 3.8 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T09:01:47.211 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:46 smithi093 conmon[33522]: audit 2021-11-20T09:01:46.216206+0000 mon.smithi093 (mon.0) 935 : audit [DBG] from='client.? 172.21.15.93:0/2483607766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T09:01:47.229 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-11-20T08:56:42.318132+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_24623": { "addr": "172.21.15.102:6825/3315610363", "addrs": { "addrvec": [ { "addr": "172.21.15.102:6824", "nonce": 3315610363, "type": "v2" }, { "addr": "172.21.15.102:6825", "nonce": 3315610363, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24623, "incarnation": 98, "join_fscid": 1, "name": "cephfs.smithi102.lhrlmh", "rank": 0, "state": "up:clientreplay", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "modified": "2021-11-20T09:01:32.855707+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 1, "stopped": [], "tableserver": 0, "up": { "mds_0": 24623 } } } ], "standbys": [ { "addr": "172.21.15.102:6827/3890204616", "addrs": { "addrvec": [ { "addr": "172.21.15.102:6826", "nonce": 3890204616, "type": "v2" }, { "addr": "172.21.15.102:6827", "nonce": 3890204616, "type": "v1" } ] }, "epoch": 102, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24649, "incarnation": 0, "join_fscid": 1, "name": "cephfs.smithi102.hjodmi", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-11-20T09:01:47.230 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T09:01:47.347 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=060abd25fc744a91823f3e1c4306cfd4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T09:01:47.350 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2021-11-20T09:01:47.362 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2021-11-20T09:01:47.408 INFO:tasks.cephadm:Teardown begin 2021-11-20T09:01:47.409 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T09:01:47.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T09:01:47.493 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2021-11-20T09:01:47.494 DEBUG:teuthology.orchestra.run.smithi093:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T09:01:47.511 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T09:01:47.529 INFO:tasks.cephadm:Stopping all daemons... 2021-11-20T09:01:47.530 INFO:tasks.cephadm.mon.smithi093:Stopping mon.smithi093... 2021-11-20T09:01:47.530 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093 2021-11-20T09:01:47.697 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:47 smithi093 systemd[1]: Stopping Ceph mon.smithi093 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:47.961 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:47 smithi093 conmon[33522]: debug 2021-11-20T09:01:47.697+0000 7f017fb80700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi093 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T09:01:47.962 INFO:journalctl@ceph.mon.smithi093.smithi093.stdout:Nov 20 09:01:47 smithi093 conmon[33522]: debug 2021-11-20T09:01:47.697+0000 7f017fb80700 -1 mon.smithi093@0(leader) e2 *** Got Signal Terminated *** 2021-11-20T09:01:48.195 DEBUG:teuthology.orchestra.run.smithi093:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi093.service' 2021-11-20T09:01:48.238 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:48.239 INFO:tasks.cephadm.mon.smithi093:Stopped mon.smithi093 2021-11-20T09:01:48.240 INFO:tasks.cephadm.mon.smithi102:Stopping mon.smithi102... 2021-11-20T09:01:48.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi102 2021-11-20T09:01:48.591 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:48 smithi102 systemd[1]: Stopping Ceph mon.smithi102 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:48.592 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:48 smithi102 conmon[36472]: debug 2021-11-20T09:01:48.362+0000 7f917a3e7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi102 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T09:01:48.593 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:48 smithi102 conmon[36472]: debug 2021-11-20T09:01:48.362+0000 7f917a3e7700 -1 mon.smithi102@1(peon) e2 *** Got Signal Terminated *** 2021-11-20T09:01:48.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi102.service' 2021-11-20T09:01:48.854 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:48 smithi102 podman[59834]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-mon.smithi102 2021-11-20T09:01:48.855 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:48 smithi102 systemd[1]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c@mon.smithi102.service: Succeeded. 2021-11-20T09:01:48.855 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Nov 20 09:01:48 smithi102 systemd[1]: Stopped Ceph mon.smithi102 for 11458b34-49df-11ec-8c2c-001a4aab830c. 2021-11-20T09:01:48.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:48.882 INFO:tasks.cephadm.mon.smithi102:Stopped mon.smithi102 2021-11-20T09:01:48.882 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-11-20T09:01:48.883 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.0 2021-11-20T09:01:49.174 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:48 smithi093 systemd[1]: Stopping Ceph osd.0 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:49.174 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:49 smithi093 conmon[49856]: debug 2021-11-20T09:01:49.033+0000 7fd928961700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T09:01:49.175 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:49 smithi093 conmon[49856]: debug 2021-11-20T09:01:49.034+0000 7fd928961700 -1 osd.0 61 *** Got signal Terminated *** 2021-11-20T09:01:49.175 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:49 smithi093 conmon[49856]: debug 2021-11-20T09:01:49.034+0000 7fd928961700 -1 osd.0 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T09:01:49.646 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:49 smithi093 podman[96015]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-osd.0 2021-11-20T09:01:49.961 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:49 smithi093 podman[96178]: 2021-11-20T09:01:50.371 DEBUG:teuthology.orchestra.run.smithi093:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.0.service' 2021-11-20T09:01:50.398 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:50 smithi093 systemd[1]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.0.service: Succeeded. 2021-11-20T09:01:50.398 INFO:journalctl@ceph.osd.0.smithi093.stdout:Nov 20 09:01:50 smithi093 systemd[1]: Stopped Ceph osd.0 for 11458b34-49df-11ec-8c2c-001a4aab830c. 2021-11-20T09:01:50.412 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:50.413 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-11-20T09:01:50.414 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2021-11-20T09:01:50.415 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.1 2021-11-20T09:01:50.696 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 09:01:50 smithi093 systemd[1]: Stopping Ceph osd.1 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:50.697 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 09:01:50 smithi093 conmon[55484]: debug 2021-11-20T09:01:50.575+0000 7f26655ce700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T09:01:50.697 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 09:01:50 smithi093 conmon[55484]: debug 2021-11-20T09:01:50.575+0000 7f26655ce700 -1 osd.1 61 *** Got signal Terminated *** 2021-11-20T09:01:50.698 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 09:01:50 smithi093 conmon[55484]: debug 2021-11-20T09:01:50.575+0000 7f26655ce700 -1 osd.1 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T09:01:50.961 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 09:01:50 smithi093 podman[96333]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-osd.1 2021-11-20T09:01:51.460 INFO:journalctl@ceph.osd.1.smithi093.stdout:Nov 20 09:01:51 smithi093 podman[96497]: 2021-11-20T09:01:51.836 DEBUG:teuthology.orchestra.run.smithi093:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.1.service' 2021-11-20T09:01:51.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:51.877 INFO:tasks.cephadm.osd.1:Stopped osd.1 2021-11-20T09:01:51.878 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2021-11-20T09:01:51.878 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.2 2021-11-20T09:01:52.154 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 09:01:51 smithi093 systemd[1]: Stopping Ceph osd.2 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:52.154 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 09:01:52 smithi093 conmon[61773]: debug 2021-11-20T09:01:52.036+0000 7f55f15ee700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T09:01:52.155 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 09:01:52 smithi093 conmon[61773]: debug 2021-11-20T09:01:52.036+0000 7f55f15ee700 -1 osd.2 61 *** Got signal Terminated *** 2021-11-20T09:01:52.156 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 09:01:52 smithi093 conmon[61773]: debug 2021-11-20T09:01:52.036+0000 7f55f15ee700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T09:01:52.461 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 09:01:52 smithi093 podman[96656]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-osd.2 2021-11-20T09:01:52.961 INFO:journalctl@ceph.osd.2.smithi093.stdout:Nov 20 09:01:52 smithi093 podman[96821]: 2021-11-20T09:01:53.300 DEBUG:teuthology.orchestra.run.smithi093:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.2.service' 2021-11-20T09:01:53.416 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:53.417 INFO:tasks.cephadm.osd.2:Stopped osd.2 2021-11-20T09:01:53.418 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2021-11-20T09:01:53.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.3 2021-11-20T09:01:53.707 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 09:01:53 smithi102 systemd[1]: Stopping Ceph osd.3 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:53.708 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 09:01:53 smithi102 conmon[43029]: debug 2021-11-20T09:01:53.574+0000 7f51b763b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T09:01:53.709 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 09:01:53 smithi102 conmon[43029]: debug 2021-11-20T09:01:53.574+0000 7f51b763b700 -1 osd.3 61 *** Got signal Terminated *** 2021-11-20T09:01:53.709 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 09:01:53 smithi102 conmon[43029]: debug 2021-11-20T09:01:53.574+0000 7f51b763b700 -1 osd.3 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T09:01:54.108 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 09:01:53 smithi102 podman[59962]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-osd.3 2021-11-20T09:01:54.417 INFO:journalctl@ceph.osd.3.smithi102.stdout:Nov 20 09:01:54 smithi102 podman[60125]: 2021-11-20T09:01:54.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.3.service' 2021-11-20T09:01:54.957 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:54.957 INFO:tasks.cephadm.osd.3:Stopped osd.3 2021-11-20T09:01:54.958 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2021-11-20T09:01:54.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.4 2021-11-20T09:01:55.168 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 09:01:55 smithi102 systemd[1]: Stopping Ceph osd.4 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:55.169 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 09:01:55 smithi102 conmon[47885]: debug 2021-11-20T09:01:55.123+0000 7f6695d31700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T09:01:55.169 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 09:01:55 smithi102 conmon[47885]: debug 2021-11-20T09:01:55.123+0000 7f6695d31700 -1 osd.4 61 *** Got signal Terminated *** 2021-11-20T09:01:55.170 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 09:01:55 smithi102 conmon[47885]: debug 2021-11-20T09:01:55.123+0000 7f6695d31700 -1 osd.4 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T09:01:55.667 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 09:01:55 smithi102 podman[60282]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-osd.4 2021-11-20T09:01:56.134 INFO:journalctl@ceph.osd.4.smithi102.stdout:Nov 20 09:01:55 smithi102 podman[60447]: 2021-11-20T09:01:56.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.4.service' 2021-11-20T09:01:56.520 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:56.521 INFO:tasks.cephadm.osd.4:Stopped osd.4 2021-11-20T09:01:56.522 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-11-20T09:01:56.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.5 2021-11-20T09:01:56.830 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 09:01:56 smithi102 systemd[1]: Stopping Ceph osd.5 for 11458b34-49df-11ec-8c2c-001a4aab830c... 2021-11-20T09:01:56.831 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 09:01:56 smithi102 conmon[52795]: debug 2021-11-20T09:01:56.678+0000 7f8e47756700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T09:01:56.831 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 09:01:56 smithi102 conmon[52795]: debug 2021-11-20T09:01:56.678+0000 7f8e47756700 -1 osd.5 61 *** Got signal Terminated *** 2021-11-20T09:01:56.831 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 09:01:56 smithi102 conmon[52795]: debug 2021-11-20T09:01:56.678+0000 7f8e47756700 -1 osd.5 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T09:01:57.167 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 09:01:56 smithi102 podman[60605]: ceph-11458b34-49df-11ec-8c2c-001a4aab830c-osd.5 2021-11-20T09:01:57.651 INFO:journalctl@ceph.osd.5.smithi102.stdout:Nov 20 09:01:57 smithi102 podman[60769]: 2021-11-20T09:01:57.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11458b34-49df-11ec-8c2c-001a4aab830c@osd.5.service' 2021-11-20T09:01:57.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T09:01:58.000 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-11-20T09:01:58.000 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11458b34-49df-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T09:02:02.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11458b34-49df-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T09:02:10.319 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T09:02:10.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T09:02:10.403 INFO:tasks.cephadm:Archiving crash dumps... 2021-11-20T09:02:10.415 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245/remote/ubuntu@smithi093.front.sepia.ceph.com/crash 2021-11-20T09:02:10.418 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T09:02:10.492 INFO:teuthology.orchestra.run.smithi093.stderr:tar: /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T09:02:10.494 INFO:teuthology.orchestra.run.smithi093.stderr:tar: Error is not recoverable: exiting now 2021-11-20T09:02:10.498 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245/remote/ubuntu@smithi102.front.sepia.ceph.com/crash 2021-11-20T09:02:10.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T09:02:10.540 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T09:02:10.542 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2021-11-20T09:02:10.544 INFO:tasks.cephadm:Checking cluster log for badness... 2021-11-20T09:02:10.545 DEBUG:teuthology.orchestra.run.smithi093:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2021-11-20T09:02:10.579 INFO:teuthology.orchestra.run.smithi093.stderr:grep: /var/log/ceph/11458b34-49df-11ec-8c2c-001a4aab830c/ceph.log: No such file or directory 2021-11-20T09:02:10.581 INFO:tasks.cephadm:Compressing logs... 2021-11-20T09:02:10.581 DEBUG:teuthology.orchestra.run.smithi093:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T09:02:10.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T09:02:10.650 INFO:teuthology.orchestra.run.smithi093.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T09:02:10.653 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T09:02:11.708 INFO:tasks.cephadm:Archiving logs... 2021-11-20T09:02:11.709 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245/remote/ubuntu@smithi093.front.sepia.ceph.com/log 2021-11-20T09:02:11.719 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T09:02:12.016 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245/remote/ubuntu@smithi102.front.sepia.ceph.com/log 2021-11-20T09:02:12.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T09:02:12.192 INFO:tasks.cephadm:Removing cluster... 2021-11-20T09:02:12.192 DEBUG:teuthology.orchestra.run.smithi093:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11458b34-49df-11ec-8c2c-001a4aab830c --force 2021-11-20T09:02:12.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11458b34-49df-11ec-8c2c-001a4aab830c --force 2021-11-20T09:02:13.207 INFO:tasks.cephadm:Removing cephadm ... 2021-11-20T09:02:13.207 DEBUG:teuthology.orchestra.run.smithi093:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T09:02:13.224 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T09:02:13.240 INFO:tasks.cephadm:Teardown complete 2021-11-20T09:02:13.240 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-11-20T09:02:13.258 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-11-20T09:02:13.259 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T09:02:13.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T09:02:13.365 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-11-20T09:02:13.385 INFO:teuthology.task.clock:Checking final clock skew... 2021-11-20T09:02:13.385 DEBUG:teuthology.orchestra.run.smithi093:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T09:02:13.389 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T09:02:13.401 INFO:teuthology.orchestra.run.smithi093.stderr:bash: ntpq: command not found 2021-11-20T09:02:13.405 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2021-11-20T09:02:13.407 INFO:teuthology.orchestra.run.smithi093.stdout:210 Number of sources = 4 2021-11-20T09:02:13.407 INFO:teuthology.orchestra.run.smithi093.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T09:02:13.408 INFO:teuthology.orchestra.run.smithi093.stdout:=============================================================================== 2021-11-20T09:02:13.412 INFO:teuthology.orchestra.run.smithi093.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 31 -6382us[-6283us] +/- 96ms 2021-11-20T09:02:13.412 INFO:teuthology.orchestra.run.smithi093.stdout:^* hv02.front.sepia.ceph.com 2 6 377 31 +1069us[+1169us] +/- 32ms 2021-11-20T09:02:13.415 INFO:teuthology.orchestra.run.smithi093.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +625us[ +625us] +/- 70ms 2021-11-20T09:02:13.416 INFO:teuthology.orchestra.run.smithi093.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 29 +2078us[+2078us] +/- 79ms 2021-11-20T09:02:13.417 INFO:teuthology.orchestra.run.smithi102.stdout:210 Number of sources = 4 2021-11-20T09:02:13.418 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T09:02:13.418 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2021-11-20T09:02:13.420 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 29 -6400us[-6190us] +/- 96ms 2021-11-20T09:02:13.421 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 2 6 377 29 +1076us[+1285us] +/- 32ms 2021-11-20T09:02:13.422 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +616us[ +826us] +/- 70ms 2021-11-20T09:02:13.423 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 31 +2077us[+2287us] +/- 79ms 2021-11-20T09:02:13.423 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-11-20T09:02:13.442 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-11-20T09:02:13.444 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-11-20T09:02:13.469 DEBUG:teuthology.orchestra.run.smithi093:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T09:02:13.530 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T09:02:13.589 DEBUG:teuthology.orchestra.run.smithi093:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T09:02:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T09:02:13.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T09:02:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T09:02:13.666 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-11-20T09:02:13.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-11-20T09:02:13.719 INFO:teuthology.task.internal:Duration was 1192.235726 seconds 2021-11-20T09:02:13.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-11-20T09:02:13.752 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-11-20T09:02:13.753 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T09:02:13.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T09:02:13.797 INFO:teuthology.orchestra.run.smithi093.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T09:02:13.800 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T09:02:13.971 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-11-20T09:02:13.972 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi093.front.sepia.ceph.com 2021-11-20T09:02:13.972 DEBUG:teuthology.orchestra.run.smithi093:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T09:02:14.031 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2021-11-20T09:02:14.032 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T09:02:14.056 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-11-20T09:02:14.057 DEBUG:teuthology.orchestra.run.smithi093:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T09:02:14.078 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T09:02:14.149 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2021-11-20T09:02:14.150 DEBUG:teuthology.orchestra.run.smithi093:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T09:02:14.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T09:02:14.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-11-20T09:02:14.344 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-11-20T09:02:14.344 DEBUG:teuthology.orchestra.run.smithi093:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T09:02:14.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T09:02:14.401 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-11-20T09:02:14.426 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T09:02:14.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T09:02:14.464 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core 2021-11-20T09:02:14.466 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2021-11-20T09:02:14.487 DEBUG:teuthology.orchestra.run.smithi093:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T09:02:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T09:02:14.538 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T09:02:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T09:02:14.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-11-20T09:02:14.568 INFO:teuthology.task.internal:Transferring archived files... 2021-11-20T09:02:14.569 DEBUG:teuthology.misc:Transferring archived files from smithi093:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245/remote/smithi093 2021-11-20T09:02:14.572 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T09:02:14.651 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6515245/remote/smithi102 2021-11-20T09:02:14.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T09:02:14.714 INFO:teuthology.task.internal:Removing archive directory... 2021-11-20T09:02:14.715 DEBUG:teuthology.orchestra.run.smithi093:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T09:02:14.719 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T09:02:14.763 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-11-20T09:02:14.780 INFO:teuthology.task.internal:Not uploading archives. 2021-11-20T09:02:14.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-11-20T09:02:14.824 INFO:teuthology.task.internal:Tidying up after the test... 2021-11-20T09:02:14.825 DEBUG:teuthology.orchestra.run.smithi093:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T09:02:14.827 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T09:02:14.843 INFO:teuthology.orchestra.run.smithi102.stdout: 262155 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 20 09:02 /home/ubuntu/cephtest 2021-11-20T15:34:08.168 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-11-20T15:34:08.201 DEBUG:teuthology.task.console_log:Killing console logger for smithi093 2021-11-20T15:34:08.209 DEBUG:teuthology.task.console_log:Killing console logger for smithi102 2021-11-20T15:34:08.210 DEBUG:teuthology.exit:Finished running handlers